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ABSTRACT 

This docuront reports on a three-phase research study 
undertaken in October 1988 for the California Association of Regional 
Occupational centers and Programs (!KX:/P). Phase 1 «fas a 
comprehensive review of the legal and fiscal development of ROC/Ps 
since their inception in 1963. Phase 2 was a cost-effects study of 
ROC/P courses, relating such variables as student enrollments, 
expenditures, revenues, and course siibjects to such variabln as 
number of students continuing, completing, and leaving, as well as 
their follow-up status. Phase 3 was che planning for the development 
of a management information system. It was found that (1) program 
planning and evaluaticai are influenced by organizational aaid 
political factors as well as by f:^scal and legal constraints; (2) 
simple cost comparisons, outcc^e analysis, and even cost~effects 
studies cannot fully describe the complex factors influencing RCX:/P 
program performance? (3) across a broad range of fiscal, 
organizational, and program dimensions, ROC/P oajiagers face rapidly 
Changing conditions and an uncertain future. In the area of fiscal 
support, data revealed year to year fluctuations as high as 30 
percent of the ioase revenues, in. the area of accountability, 
pressures for cost-effective programs had risen sharply while 
assistance and resources had improved only slightly. An extensive 
list of references is included, along with 6 appendixes that provide 
the ihstruraentation for the study.) (CHL) 
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June 22, 1989 



TOs 



Friends of RCX:/P'£5 



FROM: 



Dave Divini, President 

Ted H. Zimmerman, Chairperson, Research Committee 



RE: 



Research Study: "Quality and Effectiveness of 
California's ROC/P's" 



Enclosed for your review and information is a personal copy of the 
results and findings drawn from the research project conducted during 
1988-89 by University of California, riverside as a joint effort of 
CAROC/«^ and the State Department of Education. 

The purposes of the study are threefold: 

1. Review of the legislative development of ROC/P's with emphasis 
upon the historical enactments affecting funding. 

2. Provide a cost effectiveness analysis of selected 
courses/programs. 

3. Establish the essential framework for - Management Information 
System (MIS) to act as the basis for annual data collection and 
review of RCX:/P activities and productivity. 

The Association is deeply appreciative for the advice and assistance 
provided by all involved but in particular the study Advisory 
Committee. Their participation gave greater creditabili ty and 
assurance that the study was essential and related activities 
appropriate. 

Special gratitude is due the State Department of Education, Career 
Vocational Education Unit for the funding commitment and enthusiastic 
support of this joint research effort. 

Active participation by all the membership of CAROC/P made it possible 
to garner excellent data. A special thanks to those CAROC/P members 
who hosted the UCR research for the on-site visitations. Both were 
critical to the study - and ROC/P leaders responded! 

The results of this study will serve as a foundation for future 
Association planning and activities to improve or clarify the role of 
ROC/P within the entire California vocational education system. 
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Executive Summary 

The California Educational Research Cooperative (CERC), a research unit of 
che School of Education, University of California at Riverside, undertook in October 
of 1988 a three-phase research study for the California Association of Regional 
Occupational Centers and Programs (CAROC/P). The first phase of this study 
mvolved a comprehensive review of the legal and fiscal development of ROC/Ps in 
the State of CaUfomia since their inception in 1963. Historical records detaU a 
changmg and not always clear mission for ROC/Ps. Fiscal records over the past 
ten years showed state-level fmancing of ROC/Ps resulting in wide year to year 
vacations in the per ADA reimbursements. Furthermore, the real spending power 
of ROC/Ps has mcreased only 7.2%, compared to the entire K-12 system of 36.4%. 
The combmation of funding instability and lowered growth has created substantial 
tension for ROC/P managers. Their focus appears to have shifted more to program 
preservation and maintenance than to innovative growth and expanded service. 

Phase two consisted of a cost-effects study of ROC/P courses, relating 
pr^am "inputs" (such as student enrollments, expenditures, revenues, and course 
subject) to program "outputs" (numbers of students continuing, completmg, and 
leavmg, as well as their foUow-up status). Results showed differences in costs 
between the eight program areas studied, as well as between their completer rates. 
The rates of job placement and other positive student outcomes are relatively equal 
between coursea ROC/P finances were related to the numbers of students, with 
more students proportionaUy enrolled, completed, and placed with higher levels of 
fimdmg. Finances were unconnected, however, with hi^er rates of program 
outcomes. Instead, mcreased rates of job placement and other student outcomes 
speared to be a function of local population size, density, and wealth; the subject 

"^r.^^^}' °^^f ^^^'^^'^ unrelated to direct course operations. 
M(Kh^g ROC/P fundmg will alter the total number of students educated, 
bubstantive changes m other operational areas would be required to achie^-e higher 
rates of completion, job placement, and other positive student outcomes. 

Tbe final phase was the planning for the development of a Management 
Information System (MIS). Twenty-two different ROC/P sites were visited 
providmg ^ta on similar issues. Typically, ROC/Ps experience substantial tension 
between the busmess/economic aspects of program operation and important 
orgamzational/pohtical issues. How well sites recognize this tension and the degree 
to which they are able to dynamically respond to their environment significantK 
mfluences the ease and success of their operations. 
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This study has produced significant findings in a number of important areas. 
First, the development of California's ROC/Ps has occurred within a process that 
is dynamically responding to the changing needs of both the student clients served, 
the participating local school districts, and the State policy makers. ROC/Ps have 
evolved by serving the needs of several different important interest groups. As a 
result, program planning and evaluation are influenced by oi^anizational and 
political factors as well as by fiscal and legal constraints. 

Second, simple cost comparisons, outcome analysis, and even cost-efTects 
studies cannot fully describe the complex factors influencing ROC/P program 
performance. Sites where interdistrict harmony and home-school integration are 
dominant concerns necessarily generate a different mix of student services from 
ones where primary focus is on training adults to till Idbor market Portages. 
Moreover, differences between state-level goals and local district-level goals subject 
program managers to complex cro^ pressures. 

Perhaps the most important lesson from the research project is the extent 
to which it documents probkms of i ncertainty and instabihty confronting ROC/P 
prigram managers. Across a broad range of fiscal, organizational, and program 
dimensions, CAROC/P managers face rapidly changing conditions and an uncertain 
future. 

In the area of Fiscal Support , the data reveal year to year 
fluctuations that have reached as high as 30% of the base 
revenues. 

In the area of Regulations . CAROC/Ps have faced rapid 
changes callmg for the collection and analysis of data 
which is expensive to gather and hard to synthesize into 
meaningful program guidance. 

In the area of Accountabilitv . pressures for documentation 
for cost-effective programs have risen sharply v/hile 
assistance and resources have improved only slightly. 

In the are a of Client Recruitment . CAROC/P programs 
are confronted by the need to attract students to a sure 
program income which is controlled by enrollment in 
courses rather than by placement in jobs. 

In dealing with Cooperati ng Districts . CAROC/Ps find that 
much of their autonomy and independence is absorbed by 
preexisting teacher's contracts, limited facilities, and 
political constraints on their program options. 



Qual & EfT of CA ROC/Ps Page ii CERC i§ UCR, 6/9/89 



ERIC 



KJ 



In responding to Chantring Market Conditions . CAROC/Ps 
find themselves responsible for tracking rapidly changing 
job opportunities but have no specnlc resources or training 
appropriate to this responsibility. 

In building a Support C onstituency . CAROC/P managers 
find themsebes relatively isolated and lacking in strong 
advocaters among state policy makers, local district 
leaders, or any other m^or education interest group. 

These problems of uncertainty and instability call for substantive actions 
aimed at creating a stable environment and broad-based support for CAROC/P's 
role as a leadership agency for vocational education in California. 

If California's Regional Occupational Centers and Programs are to be fiiUy 
imderstood, any system of review and evaluation will have to take these factors 
into account. Clarifying the methods of evaluation to be used for ROC/Ps, as vvell 
as stabmzing and increasing the funding, is only part of a solution. The remainder 
comes from understanding the economic and social environment in which ROC/Ps 
exist and must function. Results from this study clearly demonstrate the 
importance of developmg a broad consensus on ROC/P goals, and the buUding of 
a Management Information System utilizing consistent data definitions and 
standardized data reduction and analysis techniques. 

r>^^f^^^ limited fiscal resources and even more limited authority California's 
ROC/Ps face a complex and difficult array of economic and political demands. The 
extent to which these demands are recognized and fully incorporated into program 
planmng and operations will dictate the; success of California's largest vocational 
education program. While many issues are addressed in this study there is a clear 
need for additional work both in policy and operations evaluation. 
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History of KOC/Ps 

To fully comprehend the status ofrepcmal ("jcupational centers and programs 
in California today one must first gain appreciation for a century and a half of 
vocational education evolution. Vocational programs have been used for a number 
of different purposes, ranging from solving high unemplo>'ment, to serving the war 
effort, to meeting the needs of special population groups. Funding and support for 
vocational education has also been mixed, vacillating from wide-spread support and 
a virtual blank check to guarded approval and limited funding. 

This section prepares the reader for the investigation that follows by 
describing the general growth of vocational education in California, and the 
emergence of the regional occupational centers and programs in particular, from the 
mid-1850's through 1988. It is important to note that as there are few works on 
this subject, much of the material on the early years is drawn from A History of 
Vocational Education in California: 1900 - 1975 by Wesley P. Smith (1979). More 
recent history is supported by the California Education Code, various state-level 
budget and operations documents, and published works. 

The Beginnings of Vocational Education {1850's to 1960*s> 

Organized efforts at vocational education in California have been recorded as 
far back as 1854 with the creation of the Mechanics' Institute "dedicated to the 
accumulation of library materials pertinent to the mechanic's craft; the 
dissemination of the lore through series of lectures; the staging of annual industrial 
exhibits; and the organization of classes in such areas as drawing, mathematics^ 
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wood carving, and metal work" ^Smith, 1979, p. 2). At that time public education 

consisted of the "grammar grades" only, since it was the common belief that 

vocational education took place on the job and not in the public schools. The 

movement towards vocationally oriented schooling began in the latter part of the 

century, with the opening of the Cogswell Polytechnic College in 1888 and the 

CJalifomia School of Mechanical Arts in 1895. 

The turn of this century saw a rapid growth of high schools, and their 

inclusion of courses in the "manual arts". Though most of the education at this 

time was tailored by the Univei-sity of California (the university having decided in 

1884 that students who graduated from accredited schools could enter the 

imiversity without further examination), there was a keen interest in the practical 

uses of a high school education. Mark Keppel, then the Los Angeles County 

Superintendent of Schools, said in a 1905 policy statement: 

The high schools of our county are doing a great work, but they are 
not doing the work they could do and ought to do. Their work is 
shaped to meet the requirements of the state university, 
notwithstanding the fact that less than one-fourth of those who attend 
the high schools can ever reach the university. The high school ought 
to fit people for life. This ought to be :he aim of the high school, and 
fitting for the university ought to be the result of this and not the 
beginning and the end of it. If the high schools are to do their duty 
to the children, they must give an increasingly large share of their 
time to commercial, industrial, and technical training. (Smith, 1979, 
p.6) 

In 1912 the state education agency was reorganized and a new position, that 
of Commissioner of Industrial and Vocational Education, was filled by Dr. Edwin 
R. Snyder. Dr. Snyder reported in 1914 on the condition of vocational education 
in California: 
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1. Vocational education was largely unorganized and had no 
uniform pattern of objectives or outcomes. 

2. A need existed to advertise the progi-am, to preach the gospel 
of vocational education. 

3. A philosophical problem existed of differentiating between 
cultural education and vocational education. 

4. Economic conditions made it imperative for youth to go to work 
early, but too little was being done to fit youth for work. 

5. A general attitude existed that vocational education was both 
narrowing and limiting opportunities. (Smith, 1979, p.9) 

Significant changes were made in the next several years to correct these 
problems. When the federal government enacted the Smith-Hughes Act (Public 
Law 64-347> in 1917, California moved quicldy to accept it. The difficulties 
associated with entry into World War One served as the impetus for the Smith- 
Hughes Act, though the effects of this legislation lasted far beyond the war. The 
main effect of this new legislation war the federal matching of state monies for 
occupational education in addition to requirements on data collection and reporting. 

Total statewide enrollment in vocational education courses increased seven 
fold in the ten year span 1920-1930. Courses were offered in the three broad 
areas designated by the Smith-Hughes Act: agriculture, trades and industries, and 
home economics. The University of California, and several teachers* colleges, began 
programs of preservice and inservice training for vocational instructors. While the 
state's policy was to integrate vocational classes into the regular high school 
setting, demand soon stimulated expansion of offerings into both new curriculum 
areas and evening class times. 

1929 saw the inauguration of the first recorded state advisory committee in 
a vocational subject (aeronautical education). The George-Reed Act of 1929 
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augmented the funding of the original Smith-Hughes legislation, allowing for an 
increase of over 20% in statewide vocational educational enrollments. New acts 
were adopted by federal legislators throughout the next several years (the George- 
EUzey Act of 1934 and the George-Dean Act of 1937) adding to the available 
federal funding for vocational education. The depression of 193U to 1940 resulted 
in a steady increase in vocational education programs. 

The war years of 1940 to 1950 moved California's vocational education back 
into the arena of providing skilled war-time laborers. The federal War Production 
Training Act of 1940 serv^ed as the impetus for much of this coursework, 
emphasizing "short, intensive instruction conducted at any time, day or night" 
(Smith, 1979, p. 28). This statute produced th.- first five year plan for vocational 
education (for 1942 - 1947) and stimulated a ne v all-time high in enrolhnents. 
Though confusion was evident following the war, enrollment quickly increased as 
the decade ended, spurred on by the funding opportunities of the G.I. Bill of 
Rights. New federal legislation, in the form of the George-Barden Act (the 
Vocational Education Act of 1946). nearly doubled the support previously provided 
for vocational programs. The net result was an increase of over 1^0 percent in 
enroUments during the ten year period, more than any other state in the union. 

Unlike the previous three decades the years from 1950 to 1960 were 
accompanied by retrenchment in both enrollment and activity. According to Smith 
a reduction in the need for laborers with specialized training, along with a booming 
economy and record growth, depressed the need for vocationally-oriented education. 
It is important to note, however, that it was during this period that Califomm first 
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studied, and then enacted, legislation allowing for the granting of high school 
program credit for work experience education. Federal legislation mo%'ed from 
broad'based funding increases to allocations geared towards particular ends, such 
as the Health Amendments Act of 1956 (which was to increase the number of 
adequately trained professional and practical nurses) and the National Defense 
Education Act of 1958 (to train "technicians" essential to "national defense". During 
the latter part of this decade an event occurred having far reaching implications. 
The issue arose in San Jose which had long operated a technical high school as a 
separate vocationally-oriented institution. Though the school was still viable and 
well supported, enrollment had declined to the point where the San Jose school 
board considered closing it. A comprehensive study concluded that the community 
would be better served by turning the facility and staff into a district wide 
vocational education center, with students from all across the district coming to the 
center for part-time training in vocational subjects. The restructuring of this 
school, and similar changes in other districts throughout the state, served as the 
seed from which California's Regional Occupation Centers grew. 

Durin^^ thp j960's vocational education both in California and the nation 
literally grew diainatically. Three pieces of federal legislation significantly 
impactcJ tlie course of vocational programs. The first was the Manpower 
DeveIopm6nt and Training Act of 1962 (Public Law 87-415). This act allocated 
federal fiinds for the training -jf hundreds of thousands of low-skilled workers 
through contracts with school districts and private schools, administered by the 
Stnte Employment Agency and the State Vocational Education Service. This 
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attempt at reducing unemployment by focusing on disadvantaged and 
undereducated individuals succeeded in training largo numbers of persons but failed 
in its original mission of reducing unemployment. The number of individuals 
seeking employment without the necessary background skills and knowledge 
continued to be unacceptably high. 

The Vocational Education Act of 1963 (Public Law 88-210) sought to 
modernize and expand the scope of the original Smith-Hughes legislation. 
Vocational education would now be available to: 

(1) Persons attending high school, 

(2) Persons who had completed or left high school and who were 
available for full-time study in preparation for employment, 

(3) Persons who were in the labor market and who needed training 
or retraining, and 

(4) Persons who had academic, socioeconomic, or other handicaps 
that prevented them from succeeding in the regular vocational 
education program. 

In addition, the Act provided for the construction of area vocational 

education school facilities, for ancillary services (such as e valuation, teacher 

training, curriculum development, experimentation, research, and leadership 

development), and for work-study programs in vocational education (Smith, 1979, 

p. 47). This was a significant departure from all prior program approaches. 

Rather than appropriating new funds along accepted program areas the federal 

government was now actwely sponsoring programs across many subject areas in a 

much more far reaching way. Smith states that The intent of the new law was 

obvious. Vocational education opportunities should become universal - in 

allocations, for all persons, and for almost all occupations" (Smith, 1979, p. 49). 
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Five years later congress pajsscii the Vocational Education Act of 196S (Public 
Law 90-576). Revisions in this law inchuled the appropriation of more money 
(slightly more than one billion dollars per year, up from $225 million in 1963), the 
creation of state advisory councils, and additional concentration jn the training of 
the handicapped, disadvantaged, and "persons who had completed or left high 
school in preparation for entering the labor market" (Smith, 1979, p.51). These 
changes helped clarify some of the pro\isions of the 1963 Act as well as providing 
substantial assistance across all areas. 



Early ROC/Ps (1963 to 1977) 

- C^fomia was quick to follow the federal lead. In 1963, new legislation was 
adopted allowing for the establishment of Countywide Vocational High Schools 
(Senate Bill 1379). This legislation created Section 7450, et d. of the education 
code which, until 1976, would serve as the 'Tiome" for all regional occupational 
education statutory language. As adopted, the stated mission of the Countywide 
High School was: 

to provide qualified students with the opportunity to attend a technical 
high school and enroll in a vocational or technical training program, 
regardless of the geographical location of their residence in a county. 
The Legislature hereby declares that a countrywide vocational high 
school will serve the state and national interests in providing 
vocational and technical education to prepare students for an 
increasingly technologicai society in which generalized training and 
skills are insufficient to prepare high school graduates for the many 
employment opportunities which require special or technical traming 
and skills. The Legislature also declares the county-wide vocational 
high schools will enable a broader curriculu? i in technical subjects to 
be offered and will avoid unnecessary dv plication of courses and 
expensive training equipment, and will provide a flexibility in operation 
which wir facilitate rapid program adjustments and meeting emergency 
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training needs as they arise. (Cal Ed Code, i$ 7450 (1963)) 
Although financial incentives were provided in the form of a permissive locaJ 
tax of fifteen cents for each one hundred dollars of assessed valuation, no sites 
were inaugurated. Wide ranging dissatisfaction with the concept of separate trade 
schools prevented their acceptance until, in 19€Z, ihe CaUfornia Legislature changed 
the name to Regional Occupational Center (ROC). Financing changes adopted at 
this time made attendance at an ROC a legal substitute for regular sc?iool 
attendance in qualifying schools for apportionment funding of general school aid. 
With these modifications in place, the first ROC site began operations during 1967. 

Amendments to Chapter 14 of the California Education Code in 1967 clarified 
the functions of an ROC as well as allowing for year-round programs and the 
inclusion of adults into the previously high-school only student base (California 
State Department of Education, 1971). Under the 1967 rules a school district or 
county ,ould form an ROC and have students from two or more high schools 
attend part-time vocational programs at the center. Since this was not always 
practical, especially in more rural locations, the mandate was broadened in 1968 to 
allow for both an ROC and a Regional Occupational Program (ROP). The ROP 
would operate hi the same way as the ROC, except that multiple sites (instead of 
a single, centraUzed one) could be used. Multi-district plans were allowed, with 
two or more single school districts jointly cooperating and enrolling students in a 
single ROC or ROP venture. Governed by local boards responsible for everything 
fix^m curriculum planning to budgetary management, three ROC/ROP 
oi^nizational types were developed: (1) single district (SD) operations, (2) joint 
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powers (JP), and (3) county operated iCO). 1969 three ROCs and fiaeen ROPs 
were in operation throughout the state. 

Rapid growth, mostly a result of the addition of the ROP provisions, moved 
the legislature in 1969 to adopt modifications aimed at more closely monitoring and 
regulating the ROC/Ps. An annual report was called for and, thour/h not directly 
funded, was produced for the next several years. Growth continued with 
enrollment in the first half of the new decade rising 474.9 percent in sixty-five 
ROC/Ps between 1970 and 1975 (California State Department of Education. 1976). 
With this explosive growth, legislative attention shifted from support and expansion 
of services to ways of controlling and maintaining program expansion. While it 
does not appear that anyone was displeased with the ROC/Ps, the resulting 
legislation was clearly intended to curb ROC/P growth. 

In 1975 Senate Bill 199 placed a limit on state-level funding to ROC/Ps not 
to exceea 105 percent of the 1974-1975 allocations (California State Department of 
Education, 1988a; Smith, 1979, p. 78). Financial records from 1974 indicate th«t, 
on the average, ROC/Ps obtained their funding from three m^ors sources: t>9.9% 
firom state ADA reimbursements (which includes federal VEA pass through jobs), 
29.3% from permissive taxes, and less than 1% from other loca' and federal sources. 
Although the ROC/Ps could have moved towards a leaner mix from state ADA 
reimbursements, thus avoiding the 105 percent limitation, they were generally 
reluctant to pursue the permissive tax. Instead, most ROC/Ps accepted the 
limitation and worked to match their enrollments to the 105 percent level. 

The next year saw a restructuring of the entire Education Code for the State 
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of California. The sections dealing with ROC/Ps were renumbered from 7450 to 
7466 to new numbers of 52300 to 52331. It was also at this time that state policy 
makers began to become less concerned with how ROC/Ps were organized and 
more interested in what results were being generated from their operation. 

The Fiscal Years (1978 to 1982) 

The five year period of 1978 to 1982 produced extensive legislative reform 
and clarification for California's ROC/Ps. New laws focused on two major areas of 
concern. The first was the ways in which ROC/Ps accomplish their mission, what 
that mission should be, and how activities would be monitored. The second area 
centered on Z/V finances. Combined, these two areas of legislative interest 
more than doubled the size of the applicable California Education Code for ROC/Ps 
over this period 

1977 saw a significant addition to vocational education by allowing regional 
occupational centers and programs to add courses providing direct work experience 
(Cal Ed Code, § 52372 (1977)). "Cooperative education" allowed the ROC/P to 
receive up to a maximum of one full day's Average Daily Attendance (ADA) 
allocation for each calendar day of student work experience. Guidelines were 
established that the ROC/P and the employer had to conform to. The reduced 
costs and practical on-the-job work experience made this one of the most popular 
and successftil vocational education programs conducted (Magnum, 1985). 

The Legislature also became interested m the outcomes from ROC/P. A 
data collection and dissemination system already in place in the Department of 
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Education was expanded (Van Ziint. 1978), with an annual report mandated from 
the Department of Education and the Board of Governors of the Community 
CoDege system This report was to review the status of all statewide vocational 
education, detailing such factors as the functioning of ROC/Ps, the numbers of 
enroUees, graduates, and job placements. Although it provided only general 
infonnation, this report began what was to become a trend toward accountability 
in statewide vocational education systems. 

1979 heralded a reversal in ROC/P funding levels. Limits were placed on 
the amount of revenue a regional occupation center or program could generate 
through its permissive taxing ability (Cal Ed Code, § 52317 (1979)). WTiile most 
ROC/P income was already coming from state ADA reimbursements (Cal Ed Code, 
§ 52321 (1979)) this change signaled that lawmakers would no longer allow 
ROC/Ps the virtually unlimited expansion available in prior years. To facilitate 
accommodation to the revenue limits imposed, ROC/P teaching time was made 
ineligible as credit towards permanent status (Cal Ed Code, § 44910 (1979)), 
essentially keeping all ROC/P teachers as temporary employees. ROC/Ps were also 
enabled to issue bonds for certain construction and capital expenses (Cal Ed Code, 
§ 52319 (1979)). 

The only significant event of 1980 was the repeal of the permissive tax (Cal 
Ed Code, § 62317 (1980)). Although this was not a significant component in most 
site budgets it did require replanning and refocusing for all of the sixty-eight 
ROC/Ps then in existence. From this point forward ROC/Ps would have to rely 
on ADA reimbursed revenue as their primary source of income. Sites were free to 
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pursue other income sources, of cour.«o (e.g., federal grants and local business 

contracts), but these sources generally proved to be of iittle impact, accounting for 

less than 10% of overall RGC/P revenue (California State Department of Education, 

1980. 1981, 1982b, 1983a, 1934, 1985, 1986a. 1987, 1988a). 

Fiscal cutbacks continued into 1981. Legislation was adopted restricting the 

amount of money ROC/Ps could recoup for student transporUtian costs. ROC/P 

reimbursements were limited to one-third of 50% of the total transportation cost. 

Moreover, this amount would only be eligible for reimbursement if the total 

transportation cost for the ROC/P exceeded 10% of the total ROC/P budget (Cal 

Ed Code, § 41852 (1981)). Optional provision was made for the ROC/?s to recoup 

part of their transportation costs from the parents of some students being 

transported (Cal Ed Code, § 39807.5 (1981)), though it is m:c]f^c.T tc what exter.t 

this possibly unpopular option was exercised by local ROC/Ps. The intent of the 

legislation, is stated in the code: 

It is the intent of the Legislature that the transportation provided by 
regional occupational centers or programs for which they receive state 
aid pursuant to section 41852, utilize, to the greatest extent possible, 
existing school buses and personnel. (Cal Ed Code, § 41852.2 (1981)) 

Naturally, the workability of such a provision depends on the cooperation of 

local school districts. No longer able to provide reimbursable transportation to 

their students on their own, ROC/Ps were forced to make a difficult decision: cut 

their budget in other areas so as to maintain the current level of transportation, 

or make arrangements with local districts and/or county agencies for transportation 

at a rate less than what self-operation might cost. Either alternatives would result 
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in increacetJ expense to the IIOC/P since neither would produce the previous level 
of reimbursement. Faced with this situation. ROC/Ps luulertook a wide range of 
cooperative efforts with participating school districts to ar»-ange for student 
transportation. 

The following year (1982) produced legislative expansion of the basic ROC/P 
mission. For the first time ROC/Ps were allowed to enroll out of school 16 to 18 
year old's, treating them essentially the same (for funding purposes) as regular 
high school studeats (Cal Ed Code, § 52314.5 (1982)). Additionally, pilot programs 
were undertaken in Los Angeles and Orange counties allow to up to ten percent 
of a site's students to come from outside the site s regular geographic service area 
(Cal Ed Code, § 52317 (1982)). An existing mandate for periodic review of courses 
was also expanded to include the input of local Private Industr>' Councils (PICs) 
(Cal Ed Code, § 52302.3 (1982)). 

A new performance report was mandated starting in 1982. Legislation (Cal 
Ed Code, § 62332 (1982)) specified that the relationship between programs costs 
and AI^A revenue Umits was to be determined for a representative sample of 
selected courses. Although mandated fur annual preparation, these reports were 
apparently only prepared for the year? 1983 and 1986. Findings from the first 
report showed a wide range of quality in fiscal accounting systems statewide with 
variations in the calculation of "direct support costs" and "indirect costs" as well as 
"charges between participating districts and the fiscal agent" (California State 
Department of Education, 1983b). These problems lead to nautions by the report's 
authors regarding the interpretability of between-site data. The 1986 report 
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reiterates the cautions of the first, while aiklinfr that education metliodology 
(cooperative education, community cla.ssroom) could not be di^stinguished and may 
confound the reported results (Calirornia State Department of Education, 1986c). 

Much of what occurred between 1980 and 1985 was closely linked to 
developments hnpacting the Ca:ifornia educational ssystem as a whole. The late 
1970*s represented a slowing of the growth of funding for all public education 
programs. The combination of a decline in enrollment, an increasing difficulty in 
passing bond elections (Wirt and Kirst, 1982), and judicial decisions such as Serrano 
V, Priest in 1974 (declaring the funding of public schools by local property taxes 
unconstitutional due to the inequitable distribution of resources) impacted ROC/Ps 
as much as the regular K - 12 sj-stem. Proposition 13, passed in 1978, "had the 
immediate effect of cutting real property tax revenues statewide by more than hair 
(Catterall and Brizendine, 1984), with the effect that local schools were now more 
than ever before dependent on state-level funding for basic operations. Rising 
standards for admission to the state's University system interacted with a 
downsizing instructional stalT to increase class sizes and eliminate or redirect 
certain instructional programs. The combined effect on California's ROC/Ps was 
not only just a reduction in available funding, but also a change in the authority 
system controlling ROC/P operations. Since the state was now in charge of the 
lion's share of ROC/P revenues (through ADA reimbursements) policy and planning 
direction shifted to the state and away from local districts. 
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A Changing Mission (19S3 to 1987) 

The only significant modification to the C'llifornia Education Code regarding 
ROC/Ps during 1983 shifted the ROC/P education clientele more towards high 
school age studenta Prior years had seen ROC/P's allowed to adjust the mix of 
high school and adult students to accommodate all types of individuals desiring 
ROC/P services. This ended when a 1983 requirement limited "growth" funding 
to support for expansion of services to 16 to 18 year old's (Cal Ed Code, § 52333 
(1983)). Some of this money was available to sites if they could demonstrate that 
they were unahle to increase 16-18 year old enrollments but did increase adult 
enrollments. The statement "assign the highest priority in services to youth from 
the age of 16 to 18 years, inclusive" was added to the code (Cai Ed Code 1983, § 
52302.5 (1983)). An annual review by each ROC/P's local governing board of 
participation from students in grades 11 and 12 was implemented, with the intent 
of preparing plans to increase participation of eleventh and twelfth graders (Cal Ed 
Code, § 52304.1 (1983)). 

This movement of ROC/Ps into the high schools continued into the next 
year with additional requirements and regulations governing the participation of 
high school students in ROC/P sponsored cooperative education (paid on-the-job) 
and community classroom (unpaid on-the-job) programs (Cal Ed Code, §§ 51762, 
51762.5, 61769, and 52317 (1984)). The pilot program started several years earlier 
in Los Angeles and Orange counties was successfully ended, with the result that 
all ROC/Ps could now enroll any number of students from outside of their regular 
service area. At the same time limits were imposed on which of the high school 
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students could be served by an ROC/ P. Previously any high school student was 
eligible (with limitations on work-experience programs dictated by state labor laws). 
Now only students 16 years or older, or at least in the eleventh grade, or having 
a special referral from a school-site counselor could participate in ROC/P programs 
and be eligible for ADA reimbursements (Cal Ed Code. § 52314 (1984)). It appears 
that the legislature was using its fui.amg prerogatives to direct ROC/Ps to service 
more and more high school students, especially those eligible to leave high school. 
One possible reason for this is an attempt to reduce the dropout rate by making 
it advantageous for the ROC/Ps to actively solicit students from a specifically 
targeted group. Unfortunately, there is no evidence of any effect other than the 
reduction of ROC/P services to hig^ school students (those ineligible under the 
new legislation) and an increase in adult ROC/P enrollments (to compensate for 
the lost revenues). 

These changes appear to be a response to a situation which had evolved over 
the years whereby essentially duplicate-fanding could be achieved for high school 
students enrolled in ROC/P vocational courses. The high school continued to 
receive full ADA reimbursement for the student while the ROC/P would receive 
an additional ADA reimbursed limit for its services. Students were not "tracked" 
into either an academic or vocational line, but rather could partake cf both. The 
legislature had moved to restrict this group of students to mostly juniors and 
seniors, though allowing each ROC/P to enroll as many of these students as it 
could support Further modifications were made during 1984 when legislation was 
passed that limited the ADA that could be earned by an ROC/P from high school 
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students (Cal Ed Code. § 52314.5 (19S4)). 

Not all problems were solved, however. ROC/Ps created in small districts 
and less populated areas were finding it extremely difficult to operate under the 
fiscal limits imposed through the prior years* legislation. The state responded to 
this need by allovdng for a greater apportionment to "necessary small ROPs"; 
namely, those managing fewer than 35C ADA in the prior year (Cal Ed Cr.de, § 
62324.6 (1985)). Using data obtained from the California Basic Educational Data 
System (CBEDS), this additional source of revenue was made available to those 
ROC/Ps who both qualified asd made special application for the monies. The level 
of additional support was initially tied to "minutes of instruction", though it was 
later modified to be based on a "minimum number of -full-time equivalent 
certificated ROP employees" (Cal Ed Code, § 52324.6 (1986)). 

The restriction of funding to ROC/Ps continued into 1986. Following the 
lead set in the Serrano decision and the desire to maintain control of education 
spending the California Legislature passed what was to become in the Education 
Code Article 1.5 - Apportionment of Funds for ROC/P (Cal Ed Code, §§ 52335.2, 
52335.4, and 62335.5 (1986)). Prior to this legislation ROC/Ps had been allocated 
fimding in much the same way as any other high school unit in California. This 
legislation had the effect of establishing a set of formulas by which the total 
revenue available to each ROC/P could be calculated. While the special funding 
for necessary small ROC/Ps was not aiTected, the base revenue for 1986 could not 
exceed that which the site had generated during 1982-83 except as allowed for by 
a percentage annual inflation adjustment in the annual Budget Act. Provisions 
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were made for growth, but were tied to a 1982-83 enrollment basis. Anticipating 
a possible budgetary shortfall, legislators included provisions*; for assigning priorities 
to funding categories and for allocating the total statewide resource among the 
operating ROC/Ps. 

These calculations were modified somewhat in 1987 (Cal Ed Code, §§ 
52335.25, 52335.4, and 63225.6 (1987)), but the intent had become clear. No longer 
could California's ROC/Ps expand at any desired rate and still receive the same 
level of support for all students. Scarce state resources had to be distributed fairly 
and equitably across the state. A new report was commissioned to prepare findings 
on these actions and to make recommendations concerning the efTectiveness in 
achieving statewide equity in the allocation of funds and ADA (Cal Ed Code, § 
62335.8 (1987)). 

Financial records from this period indicate that the net effect of this 
legislation was reduced growth and a concurrent stabilization of funding to ROC/Ps, 
with only minor increases allowed in the annual Budget Bills. This resulted in a 
moderate expansion of the spending power of ROC/Ps when adjusted for inflation 
and total ADA served, increasing 7.2% in the period from 1978 to 1987. This 
ejqpenditure growth was quite small when compared with the 35.4% increase in 
total education funding during the same period. ROC/Ps funding increased only 
about one-fifth as much as the average of all state K - 12 public programs. Placing 
constraints on fiscal resources did not, however, stabilize the programs or mission 
of the ROC,/Ps. Rather, they encouraged ROC/Ps to approach both . avenue 
devfclopment and expense-reduction on an ad hoc, expediency basis. 
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The adoption of the Hughes-Hart Kdueational Reform Act of 1983, commonly 
known as SB 813, contributed further to the ROC/P s change of mission. This new 
legislation forced schools into increasing the number of academic courses offered. 
Three years after passage of SB 813, Nowak and Hiatt (1986) reported an overall 
increase in academic offerings of an average of 15.4% statewide. This was 
accompanied by a decrease in vocational education offerings in the high schools by 
an average of 15.9%. The new graduation requirements mandated by SB 813 
reduced the number and variety of elective courses available to high school 
students, with the effect of completely eliminating or drastically reducing the high 
school vocational offerings. These authors conclude that, while some students will 
be better prepared to enter college as a result of SB 813, more students may 
become dropouts who are less prepared to enter the world of work due to the 
decrease in the vocational education options. 

Current Trends (1988) 

In 1987-1988, the year covered in this research, California's ROC/Ps found 
themselves at the center of crucial fiscal and organizational issues. Funding 
increases, readily available in previous years, have slowed when compared to other 
state supported educational programs, becoming less responsive to the demands of 
either growth or inflation. Furthermore, since the Legislature decides on the bulk 
of ROC/P allocations, ROC/Ps must be as responsive to alliances in Sacramento as 
in their own community. 

Increased attention to concurrent high school students, continued with 
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increased academic requirements in SB 813, have encouraged ROC/Ps to offer 
many vocational courses high schools are no longer able or willing to provide 
(Nowals. 1986). Recent funding restrictions make it difficult for ROC/Ps to meet 
these revised goals, however. Unable to afford expensive centers and saddled with 
unreimbursed transportation costs, most ROC/Ps must work diligently to establish 
working alliances with local school districts and county agencies in order to obtain 
students, facilities, ard equipment. The requirements of SB 813 complicate 
development of strong alliances by moving further from a vocational emphasis than 
ever before. High school policy makers are ready to divest themselves of the 
responsibility for providing vocational courses, and reluctant to encourage vocational 
programs that threaten to divert students from mandated SB 813 objectives. 

ROC/Ps are also facing a growing interest in system-wide accountaoility. 
Recent legislation (Cal Ed Code, §§ 8007 and 52302.3 (1987)), requires ROC/Ps to 
prove their worth as purveyors of skm based training. This, despite evidence that 
skiH training per se is not as important for new employees as is positive work 
habits and attitudes (Wilms, 1983). Earlier accountability measures were largely 
internal, centering on how well ROC/Ps delivered stated services. Course review 
and approval procedures, as well as the two-year program review, were ROC/P 
quality control mechanisms. Newer legislation and interpretation by the California 
State Department of Education, (1987a, 1988a) shifted the emphasis away from 
product delivery to subsequent successful placement of students in the job? for 
which they are trained. The latter objective does not necessarily conflict with the 
former, but they could lead to different program development and implementation 
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strategies. In the fir^t case ROC/Ps are encouraged to deliver a course so as to 



maximize student performance in the course material. In the second case the 
ROC/P is encouraged io screen out potentially unsuccessful students and to retain 
only the most promising students; training them in skills directly needed for initial 
emplo3mient. 

The mission of regional occupational centers has grown significantly from 

what was first approved in 1963. Today, that mission is: 

to provide qualified students with the opportunity to attend a technical 
school or enroll in a vocational or technical training program, 
regardless of the geographic location of their residence in a county or 
region. The Legislature hereby declares that a regional occupational 
center will serve the state and national interests in providing 
vocational and technical educational to prepare students for an 
increasingly technological society in which generalized training and 
skills are insufficient to prepare high school students and graduates, 
out-of-school youth and adults for the many emplojonent opportunities 
which require special or technical training and skills. The Legislature 
also declares that regional occupational centers will enable a broader 
curriculum in technical subjects to be offered, and will avoid 
unnecessary duplication of courts and expensive training equipment, 
and will provide a flexibility in operation which will facilitate rapid 
program adjustments and meeting changing training needs as they 
arise. 

It is recognized by the Legislature that vocational programs may 
achieve great flexibility of planning, scope and operation by the conduct 
of such programs in a variety of physical facilities at various training 
IcKnitions. 

It is the further intent of the Legislature that regional occupational 
centers and program provide vocational and occupational instruction to 
the attainment of skills so that trainees are prepared for gainful 
employment in the area for which training was provided, or are 
upgraded so they have the higher level skills required because of new 
and changing technologies or so that they are prepared for enrollment 
in more advanced training programs. (Cal Ed Code, f 52300 (1988)) 

ROC/Ps now serve both high school age persons, whether in a high school 
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or not, as well as other adults seeking vocational training. Previously held 
accountable for "vocational and occupational instruction related to the attainment 
of a specific skill which will make the trainee technically equipped to be 
immediately employable" (Cal Ed Code, § 7450 (1969)), ROC/Ps today are 
responsible for training for the acquisition of employment, upgrading of skills, and 
preparation for enrollment in further training. Instead of serving as the final step 
in a high school vocational education process, ROC/Ps today are assuming much, 
if not most, of the primary high school pre-vocationa; education mission. 

Conclusions 

Historical review has shown California's regional occupational centers to be 
an educational delivery system afTected by a number of competing forces. 
Established as an outgrowth of available federal funding in the early 1960's, the 
ROC/Ps have evolved to an organization with a complex and multi-faceted mission 
and relatively scant resources with which to accomplish it. Five distinct threads 
wind their way through this history, each contributing significantly to the 
operations of ROC/Ps. 

The first is concerned with the way in which ROC/Ps are financially 
supported. ROC/Ps are now almost entirely dependant on state ADA 
reimbursements for operational funds. The permissive tax provided in early 
legislation was rarely employed and was eventually repealed Unlike other public 
schools, there is neither a mandate for attendance nor a minimum funding level 
available to ROC/Ps. Instead, ROC/Ps have always had to attract potential 
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students into thviv programs. Since enrollment is voluntary, ROC/Ps must actively 
recruit students in order to maintain income levels in a reimbursement system 
based on student attendance reimbursement (ADA). 

Initially ROC/P vocational programs were open to virtually any student 
who expressed intent. This lead to an explosive growth in ROC/P offerings, and 
state reimbursements, which the state legislature curbed by capping enrollments, 
limiting growth, and restricting the kind of student that could enroll. Further 
I^islative mandates provided incentives to the ROC/Fs to enroll greater numbers 
of high school students, relieving the high schools of a substantial portion of their 
vocational education responsibilities. This was agreeable with many high schools 
seeking relief from increased graduation requirements embodied in SB 813. The 
ROC/Ps provided a natural agency to take over the vocational education needs of 
these students. 

By accepting such an expanded mission the ROC/Ps moved further into the 
high school environment. In doing so they accepted responsibility for pre-vocational 
education instead of the upper level vocational training previously held. Such an 
arrangement seemed to insure that the ROC/Ps would have a steady flow of new 
students as well as readily available facilities (and, in some cases, equipment). 
However, it flso put them undei the indirect control of the local high schools. 
With the high schools able to control the ROC/Ps primary income stream (by 
mediating student access to the ROC/P programs) high schools were able to dictate 
an increasing share of ROC/P operations. Some ROC/Ps were viewed less as an 
alternative educational entity and more as a subdivision of the local district(s), with 
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cost-efficiencies being determined at the district, and not ROC/P, level. 

Although never stated outright, this produced a subtle yet powerful shift in 
the true mission of the ROC/Ps. Initially intended to provide quality training for 
job readiness to a wide mix of students, ROC/Ps became more and more concerned 
with district and county organizational affiliations in order to support continued 
high school student enrollment. A good ROC/P became one that could deliver 
educational services m such a way as to benefit the participating school districts. 
This was typically measured not by student job placements but rather by students 
enrolled into the ROC/P program and, therefore, out of the high school for that 
time during the day (leaving the high school additional regular seats). Not 
insignificantly, they also became sources of increased income for the local district(s). 

The final lesson from the ROC/P history is the prominence of state 
legislators' concern that ROC/P funding produces little beyond the benefits already 
resulting to high schools and local districts/counties. ROC/Ps were initially 
required to detail the costs of their operations and, more recently, to show what 
outcomes are being achieved. Unfortunately, these outcomes address only specified 
job placements ignoring both upgrade skills training (part of the legislative intents 
for ROC/Ps) as well as the number of high school programs shifted to the 
ROC/Ps. Reports generated from such data therefore only show a portion of 
ROC/P outcomes, and not necessarily those with which ROC/P managers are most 
concerned. Currently, funding for ROC/Ps rests not on their ability to either 
provide a quality educational service or success at training and placing students in 
particular vocational fields. Rather it is their ability to attract and secure students 
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from within a limited population group that controis revenues. Cooperation with 
local high schools is essential, since that is where most potential students are to 
be found and where important facilities and equipment for the training are located. 
Mth limited growth, and insufficient funds to be truly independent of the high 
school, ROC/Ps must work to ensure adequate enrollments and income. On top 
of these constraints, however, the state is requiring performance reporting limited 
to only a small range of the actual outcomes produced. 

If it is the intent of the legislature to use ROC/P fiscal support to control 
program performance it must be demonstrated that, in the current operational 
environment, fiscal resources are related to program outcomes. This study 
examines that issue in detail Should such a relationship not be found (as it 
appears with these results), ROC/P evaluation strategies will have to be 
reconsidered. Since California's regional occupational centers and programs must 
pursue a number diverse objectives, using competing strategies for their 
achievement, it is vitally important to identify which organizational and program 
characteristics contribute most to success in meeting various goals. 
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Fiscal Deveiopmen is 



Prior to 1969-70, State fiscal reports do not ineiiide any information 
specifically relating to ROC/Ps. Detailed fiscal data do exist for the ten-year span 
from 1977-78 through 1986-87. Analysis of these data provide a clear picture of 
ROC/P fiscal development. ROC/P income is largely controlled by attendance 
rates. Between 1977-78 and 1986-87, the average daily attc.idance (ADA) of Single 
Di?trict/Joint Powers ROC/Ps increased from 35,137 to 53,577, more than 60 
percent. The ADA of County-Operated ROC/Ps, first reported separately in 1980- 
81, increased from 33,838 to 48,947. Total ADA (County level ROC/Ps plus Single 
District/Joint Powers ROC/Ps) increased from 70,829 in Fiscal Year (FY) 1977-78 
to 102,524 in FY 1986-87. Overall this represents an approximately 45 percent 
increase in ADA 



Year 



S.D./J.P. 
35,137* 
33,220* 
38,962* 
38,085 
40,480 
47,195 
39,267 
46,400 
49,384 
63,577 



CO. 



Total 
70,829 
56,300 
63,112 
71,923 
82,183 
91,456 
77,516 
90,981 
96,197 
102,524 



1977-1978 
1978.1979 

1979- 1980 

1980- 1981 

1981- 1982 

1982- 1983 

1983- 1984 

1984- 1985 

1985- 1986 

1986- 1987 



33,838 
41,703 
44,261 
38,249 
44,581 
46,813 
48,947 



] Data from California Controller's Office 

'* Data from Office of Local Assistance, Sacramento 



Table 1; Annual Average Daily Attendance 
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Regional Occupational Center/Program 

Annual Average Daily Attendance 




+— statewide Total 



Total income for ROC/Ps was S94.9 million in 1977-78. rising to $251.4 
million in 1986-87. 





S,D./J.P. 


CO. 


Total 


1977-1978 


48.6 


46.3 


94.9 


1978-1979 


47.1 




89.0 


1979-1980 


53.2 


27.5 


80.7 


1980-1981 


61.0 


66,7 


127.7 


1981-1982 


69.8 


77.6 


147.4 


1982-1983 


84.1 


81.9 


166.0 


1983-1984 


89.5 


87.5 


177,0 


1984-1985 


96.6 


90.9 


187.5 


1985-1986 


113.4 


105.7 


219.1 


1986-1987 


138.6 


112.8 


251.4 



' in Million^ of Dollars 



Table 2: Annual Income* 



Although this appears to be a very sizeable increase, this income is in actual 
dollars, not adjusted for inflation. To insure between-years comparability the 
Implicit Price Deflator (IPD) was used as a measure of inflation. This general 
indicator of overall economic growth in the nation increased slightly more than 70 
percent m the ten year span examined, from 140.05 in 1977 to 239.11 in 1987. 
When ROC/P income is adjusted for inflation using the IPD the increase in income 
is approximately 65 percent over the decade of interest, rising from $67.76 million 
to $106.14 million. These data are presented in Table 3 and jointly in Figure 2 
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Year 




GNP/IDK" 


Adi T<>tnr 


1977-1978 


94 9 


140 05 


fi7 7fi 


1978-1979 


89.0 


150 42 


59 17 


1979-1980 


80.7 


163 42 




1980-1981 


127.7 


178 42 


71 57 


1981-1982 


147.4 


195.60 


75.36 


1982-1983 


166.0 


207.38 


80.05 


1983-1984 


177.0 


215.34 


82.20 


1984-1985 


187.5 


223.43 


83.92 


1985-1986 


219.1 


237.45 


92.27 


1986-1987 


251.4 


239.11 


105.14 



* in Millions of Dollars 

* Base of 1972 = 100.00 



Table 3: Annual Income Adjusted for Inflation 

Increased ROC/P income was also adjusted for the increasing size of the 
student population served, as measured by ADA. 



Year 


Adi. Income" 


1977-1978 


67.76 


1978-1979 


59.17 


1979-1980 


49.38 


1980-1981 


71.57 


1981-1982 


75.36 


1982-1983 


80.05 


1983-1984 


82.20 


1984-1985 


83.92 


1985-1986 


92.27 


1986-1987 


105.14 



* in Millions 

* in Dollars 



Total ADA 


Inc/ADA 


70,829 


956.67 


56,306 


1,050.86 


63,112 


782.42 


71,923 


995.09 


82,183 


916.98 


91,456 


875.28 


77,516 


1,060.43 


90,981 


922.39 


96,197 


959.18 


102,524 


1,025.52 



Dollars 



Table 4: Annual Income per ADA in Adjusted Dollars 
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Regional Occupational Center/Program 

Statewide Total Annual Income 




1979 



1981 1982 1983 1984 1985 1986 

Year 



Unadjusted Dollars 



Adjust for Inflation 



to 

O 



> 
S 

e 



3 

1987 i 



43 



In FY 1977-78 the average income (in inflation-adjusted dollars) per unit of 
ADA was $956.67. In FY 1986-87 the adjusted income per ADA was $1025.52 per 
unit, a net increase of $68.85 ( 7.2^e). Wide variations in both enrollment and 
income plagued ROC/Ps throu^out the decade, however. Per ADA income varied 
between a low of $782/ADA in FY 1979-80 and a high of $1060/ADA in FY 1983- 
84. The low figure probably reflects the overall impact of ^reposition 13 on 
schools in California 

While the decade long picture shows significant enrollment growth, total ADA 
for aU ROC/Ps actually decreased from 91,456 in FY 1982-83 to 77,516 in FY 1983- 
84. It is not clear why the ADA suffered such a decline, particularly since the total 
ADA rose to almost the FY 1982-83 level in the next year (FY 1984-85). These 
data are presented in Figure 3. 

While ROC/P funding has increased over this ten year span the increase 
has neither been dramatic nor particularly stable. This instability in available 
revenues, coupled with the slight increase in spending power, and a renewed state- 
level interest in linking funding to performance criteria represent the significant 
points for this portion of the research. 
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Regional Occupational Center/Program 

Income per ADA in Adjusted Dollars 




760 



.J 1 



1978 1979 1980 1981 



1982 1983 1984 

Year 



... 1. 

1986 



1 

1986 



1987 2. 



^ Statewide Total 



Ciwt-Effects Analysis 

A cost-outcome analysis of an educational progi*am is a method whereby an 
attempt is made to relate factors needed to operatp the given program with the 
outcome (or outcomes) that the program produces (Bentkover, Covello, & 
Mumpower, 1986). Methodologies for determining the inputs required to produce 
given levels of output are not new. Smith and Smith (1983) cite over 350 recent 
studies which relate costs to outcomes in a number of different fields, from 
medicine to education to politics. Before the relationship between costs and 
outcomes can be examined, though, it is necessary to gain a dear understanding 
of the meaning of each term. 

Brinkman and Allen (1986) present the most cogent discussion of calculating 
the costs of a given program under study. Essentially, they state that there is no 
single, universal definition of costs. Rather, the cost of an enterprise depends 
greatly on the context in which the costs analysis question is being asked- They 
describe several ways in which costs may be considered: 

* Objectives (input, output, activity, organizational unit 

* Basis (historical, projected, standard, imputed, replacement) 

* Assignability (direct, indirect, full) 

* Variability (fixed, variable, semi-variable) 

* Activity relationship (total, average, marginal) 

* Determination method (specific service, continuous service) 

* Tune relationship (time period, w crual or cash, deflated) 

Additionally, program costs are differentiated depending on whether the costs 
are recurring or nonrecurring as well as controllable or noncontroUable. Several 
authors have wrestled with this issue of what costs ought to be computed and 
included in a model. Both Nagel (1983) and Catterall (1982) argue for including 
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the cost of nonmonetary items. Smith (1984a) distinguishes between costs 

generated as part of a project's start-up versus those generated as a result of a 

program's operations. In a similar light, Thompson (1980) lists five different 

categories for computing costs (measurement, valuing, econometric methods, 

hypothetical questions, and political choices), while Levin and Seidman (1981) 

define five different valuation techniques (market prices, shadow prices, joint costs, 

annualized costs, and present value). 

In the actual process of collecting and tabulating cost information, I-evin 

(1983) advocates the use of an "ingredients model" for describing program costs. 

Essentially, every aspect of the program is enumerated with a cost (even if zero) 

assigned to each. He states: 

Basically, the ingredients model will require that we specify all of the 
ingredients that are required for any particular intervention. Once 
these ingredients are specified, a value is placed on each of them, 
aevin, 1983, p. 49) 

The determination of what that value will be is just as important as knowing 

that a cost must be assigned for a given component. He argues that the cost used 

is not necessarily the price paid, but rather that 

. . . the cost of a specific intervention will be defined as the value of 
all of the resources that it utilizes had they been assigned to their 
most valuable alternative use. In this sense, all costs represent the 
sacrifice of an opportunity that has been forgone. By using the 
resources in one way, we are giving up the ability to use them in 
another way, so a rost has been incurred, (p. 48) 

It is obvious that even contributed or donated resources such as 
volunteers must be included as ingredients according to such an 
approach, for such resources will contribute to the outcome of the 
intervention, even if they are not included in budgetary expenditures, 
(p. 52) B J 
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Essentially, under Levin's scheme every aspect of a program must be 
assigned a cost value and be included in the total cost determination for that 
program. Determining what is the "most valuable alternative use", and what that 
alternative use might be worth, is no short order. Researchers must agree on boUi 
how this is done and what value is assigned in order for the results of any two 
studies to be comparable. 

Not all researchers agree with Levin in this regard. Haggart (1978) 
differentiates costs on the basis of how the results of the study will be used He 
argues that, for studies whose results are to be used within a single operational 
unit (such as a school district) it is necessary only to collect "district specific" cost 
items. When the study is to be generalized outside of the unit (district), costs 
representing a fuller range of items should be included Smith (1985), in a work 
on streamlining the data collection techniques in cost-outcome analysis, clarifies 
this: 

To streamline the calculation of program costs, an intermediate step 
can be added identifying resources that represent direct costs to the 
sponsor. If the evaluation question pertains to the direct cost assumed 
by the sponsor, the calculation of program cost should include onjy 
those resources that involve direct costs. . . . This generally does not 
require the calculation of the value of resources such as volunteer 
time. . . . Such inclusion of opportunity costs would spuriously elevate 
the total cost figure for local analysis purposes. If a program is an 
acyunct to regular school programming, then only those costs over and 
above the regular school costs need to be included, (p. 7) 

This differentiation of purposes is also magnified when the issue of "costs 

already taken" is addressed. Writ£**s on this point seek to understand whether it 

is appropriate to include costs for those goods and services incur, ed prior to the 
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start of the program in question. Dunn and Suilins (19S2) take a social welfare 
approach in discussing costs in a study on higher education by seeking to identify 
all costs, both monetary and nonmonetary, that society must bear to conduct a 
program. While counter to that of Smith, it essentially states two positions, one 
which advocates costing on the basis of total program inputs (direct, indirect, and 
support by society at large) while the other advocates costing on the basis of 
differences between programs (or between a program and no program at all). 

Determination of program outcomes is likewise dependent on the context in 
which the program is held. Both Smith (1984b) and Hanson (1986) state the 
necessity of quantifying the results from the program in order to be sure that the 
comparisons are both interpretable and valid. Outcomes, though, can be both to 
the program (such as program continuance), to the providers (continued fimding), 
and to the clients served (such as a particular skill learned or employment gained). 
In fact, Smith (1984b) recognizes the strong possibility that single programs may 
have multiple, and quite varied, outcomes. In such cases the various outcomes 
may be comparable within themselves but not in-between, leading to a need for 
differing methodologies if such a comparison is necessary. 

Using these definitions of costs Levin (1981, 1983) has organized the two 
concepts of costs and outcomes together in evaluating educational programs. He 
describes four different kinds of cost-outcome analysis: cost-feasibility, cost-utility, 
cost-benefits, and cost-effects. Each method addresses a particular combination of 
costing and outcome problems as well as contextual frameworks in such a way tiiat 
an evaluation could be performed. 
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The simplest kind of co^t-oiilcoino evaluation is lh«' '-ost-feusibilitv (CF) 
analysis (Smith and Smith, 19S3). In thi^ situation there is a known and finite 
amount of resource that may be invested into a given project The total costs for 
different programmatic alternatives for the project are tabulated and compared. 
Any program whose total cost is more than the available resource is automatically 
eliminated, since it would be impossible (given the fmiteness of the resource) to 
meet its costs. All of the remaining less-or-equal costly programs are then ordered 
from least expensive to most and, all other things being equal, the least expensive 
program is chosen. The assumptions are made that no more than the finite 
resource may be expended on the project (regardless of the alternative), and that 
each program possibility ^tzxids an '^qual chance of achieving the same level of 
outcome. When these conditions are both known and applied CF analysis 
represents the most strai^tforward method for deciding between program 
alternatives. In essence, this approach relies on cost differences between similarly 
producing programs for decision making. 

The next level of complexity in cost-outcome analysis is that of cost-utility. 
This type of analysis is useful when costs can be quantified (according to some 
method), but outcomes are varied and may only be estimated. In these cases the 
value of the outcome (its utility) is not completely known and may have to be 
determined in part, if not in full, by more subjective methods. (^st-utiUty (CU) 
analysis permits the use of both quantitative and qualitative data, including that 
of different measures of outcomes for the different programs under evaluation, by 
forcing the user to weight the values of the responses according to their perceived 
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impact, then comparing those weights according to the costs determined (Manpower 
Administration. 19S3). Unfortunately, CU's flexibility is also its drawback, since 
differing outcomes measured qualitatively may produce different ratings (and, 
indetd, ev^n liifferent systems of rating) from person to person. Thus, the mei;hod 
could be found to be highiy subjective, with the results useful only within the 
group doing the evaluation. 

levin's third method of analysis, stemming from a social welfare point of 
vidw. seeks to overcome the limitations of CU analysis. Rather than treating th*? 
outcome subjectively, cost-be.iefit (CB) analysis seeks to assign a monetary value 
(or worth) to each outcome. Each outcome is, therefore, carefully examined for 
both its short-term and long-term impacts on both the clients, the sponsor, and 
society at large. Positive programs are those which have a high raoneiaty value 
to the outcome relative to the monetary value of the program cost (Hall, 1976). 
Dunn and SuUins (1982) put this slightly differently: 

If one accepts the assumption that public resources should be invested 
so social welfare is maximized, then cost-benefit analyses should be 
used to identify investments that will achieve that result, {p. 21) 

While interesting in theory, the problem of assigning costs to the long-term 
products of different outcomes can be both involved and confusing. Many authors 
(Thompson. 1980; Dunn and Sullins, 1982; Barnett, 1985; Levin tmd Seidman, 1981; 
and Hunting, Zymelman. & Godfrey. 1986) acknowledge this problem, though it is 
Dunn and Sullins (1982), in a study of the outcomes from higher education, who 
pat it most succinctly. 

In summary, estimating the value of the social return on higher 
education is an imprecise process. Me. iuring the earnings differential 
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of higher education is, in itself, difilcult given the many other factors 
which are also highly correlated with one's Income level. 
Nonmonetary direct returns to higher education are less easily 
measured. Iridirect monetary and nonmonetary returns are even more 
difScult to measure. Hence» arriving at a precise and defensible value 
probably is not possible. Once again, the utility of cost-benefit analysis 
is challenged seriously by the tenmus character of one of its key 
components, (p. 30) 

Cost-benefit analy sis, while it does address the shortcomings of cost-utility 
analysis, has its own problems which may severely impact its usability. When both 
short- and long-term outcomes can be clearly identified and reliably collected, the 
value of CB analysis in addressing issues of broad social concern is unparalleled 
(Escobar, «^rnett, and Keith, 1988: Dean and Dolan, 1986). Without those 
assurances, though, it is a process (like cost-utility analysis) that is open to 
criticism. 

The final method of cost-outcome analysis forwarded by Levin (and others) 
is tLat of cost-effects (CE) analysis. Cost-effects analysis incorporates, as do the 
other methoos of analysis, direct costs of program operation. Outcomes, however, 
are assumed to be both similar between the programs and measurable in 
quantitative terms. Since both costs and outcomes are expressed in quantitative, 
numerical tenns it is possible to forni a ratio between the two, known as the 
cost/efTects ratio. Levin describes this ratio: 

In order to use the CE approach, it is first necessary to determine the 
program objective and an appropriate measure of effectiveness. Then 
the alternatives that will be evaluated must be specified. Given these 
requirements, it is possible to design an evaluation of the alternatives 
on the particular criterion of effectiveness that has been established 
and to obtain cost information for cBch alternative. Finally, the cost 
and effectiveness data can be combined into CE ratios that show the 
amount of effectiveness that can be obtained for an estimated cost. 
Since these ratios can be compared among alternatives, it is possible 
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to provide information to decision makers on which alternatives seem 
to be most parsimonious in terms of costs, with respect to the measure 
of effectiveness that is under scrutiny, (p. 115) 

Numerous studies have been undertaken in education using cost-effects 
analysis. Quinn. Mondjfrans, and Worthen (1984) report applying CE techniques in 
a multiple regression analysis in order to identify the significant factors of the CE 
ratio in a study comparing different methods of mathematics instruction. Levin 
and Meister (1986) used this method in an in' 3tigation of four different methods 
of instruction in teaching math and reading (see also Chandler, 1986). Both the 
Technical Assistance Centers (Estes, 1983) and certain reading and math programs 
(Fitzgerald and Hunt, 1985) established under Chapter I funding were evaluated 
using CE methodologies. 

Unfortunately, the literature also contains examples of studies calling 
themselves cost-effects when, in reality, the researchers only studied the 
comparative costs of the different programs (Dyer, 1985 being a representative 
example). These studies confuse the concept of cost-effects as defined above with 
that of cost-efficierjy or, namely, finding the least expensive program. Such 
evaluations are more in kind with cost-feasibility studies than cost-effects studies, 
ance little to no attempt is made to relate programmatic costs with 
programmaticaJly-induced outcomes. The four methods of cost-outcome analysis are 
summarized in Table 5. 

In general. Levin (1983) and others already cited (Smith and Smith, 1983; 
Smith. 1984a, 1984b. 1985; and Dunn and Sullins, 1985) find the cost-effectiveness 
framework to be the one more suited to most kinds of educational evaluation. The 
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major drawback of CB analysis is that it does not t\pically take into account the 
social value (or return) of the program. Nonetheless, it represents a method of 
comparison generally more stringent and repeatable than either cost-feasibility or 
cost-utility. When comparing two (or more) programs attempting to achieve the 
same (or similar) results and the overall operation of the programs are not at 
question (such as in CB analysis), it appears that cost-efTects analysis is the cost- 
outcome method of choice for most educational program analyses. 



Typ e Costs 
CF Program 

CU Prog/Soc 

CB Social 



Outcomes 
Similar 



Strengths 
Easy to use 



Weaknesses 

Essentially cost analysis 
only 



Can be varied Less stringent Difficult to reproduce; 

highly subjective 



Social impact Economic valuing of 

outcomes 



Long-term 

CE Program Similar & specific Fully quantitative Enforces similar outcomes 



Table 5: Comparison of Cost-Outcome Evaluation Methods 



Limitations of Previous Research 

The interest group for this investigation is comprised of California's seventy 
regional occupational centers and programs; therefore, the literature relating to 
cost-outcome analysis in vocational education for the state of CJalifomia has also 
been investigated. While the intent is to highlight only those sources dealing with 
ROC/Ps directly, a number of other interesting works were found relating to other 
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vocational education programs found throughout the state. 

Basic information on vocational education in California during the last few 
years is found in a number of different places, with the most prominent agency 
being the California State Department of Education (1986b, 1986d, 1986e, 1987, 
1988b). Vocational education in California consists of separate programs, such as 
through the ROC/Ps, in addition to programs integrated into the high schools and 
community colleges. The focus over the last few years, however, has been a 
movement towards a tighter infusion of these vocationul education efforts into the 
mainstream of the K - 12 program. This is best exemplified in the 1986 
Department of Education Annual Report (California State Department of Education, 
1986d) which states: 

California is in a period of transition - changing technology, changing 
jobs, and a dramatically changing population. The education and 
training of our students must change, as well, to remain relevant for 
the future. Many more of our students than ever before, regardless 
of career aspirations, now require a strong academic foundation to be 
successful. Academic and career-vocational preparatic;i can no longer 
function as mutually exclusive enterprises. Instead, they must function 
as complementary components of a total process, (p. 27) 

Movement towards this end is taking a number of forms. One is as a result 

of Assembly Bill 3639 (Chapter 1138, California Statutes of 1986) which authorizes 

appropriations to study the "2 + 2" and + 2 ^ 2" programs. These programs 

encourage high school districts, regional occupation centers and program, the state's 

community college system, and the California State University and University of 

California to work together to provide articulation agreements for students in 

specially designed vocational career programs. Under these articulation plans 
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students would begin their vocational education in the high school, progress 
through an ROC/P and/or community college towards an Associate's degree, and 
then on to a four-year school for a Bachelor's degree. 

Quality indicators, one of the fall-outs from the SB 813 movement for 
accountability in education (Hobson, 1983; Fetler, 1986), are also being stressed in 
vocational education circles. While vocational administrators generally support the 
notion of accountability in education, there is little consensus as to how that 
accountability both ought to be measured and how changes can be effected to 
improve it. Dual-track plans such as those adopted in some European countries 
hold much promise for achieving such accountability (Werner, 1986), since students 
could then be compared against other of similar interests and aptitudes. There is 
little evidence that California would move in such a direction, though, as current 
literature points to an ever-increasing integration of all education, career-vocational 
and academic, under a single roof. How this will affect the determination of 
vocational education's performance is unclear. 

The best orientation to California's system of regional occupational centers 
and pn^ams can be found in their Operations Handbook (California State 
Department of Education, 1988). This document serves as a working blueprint that 
current ROC/P administrators can use as a guide to both day-to-day operations and 
long term planning. As such, it provides a unique orientation to the current 
operational outlook on cost-outcomes analysis within the or^nization. 
Unfortunately, while this document repeatedly states the need for performance 
analysis and cost-effective operations, there is little in the way of concrete material 
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on which a cost-effects study could be based. Specific fiscal data, student 
enrollment and attendance figures* and follow-up reports are cited, however, which 
could serve as a basis for a study in the area. 

This is not to say that no prior studies have been accomplished. Indeed, 
several works regarding ROC/Ps (both theoretical and applied) have been 
performed over the last few years. The earliest of these studies was done in 1971 
as a result of a change in the education code mandating such (Cal Ed Code, § 7463 
(1969)). This report summarizes the historical development of ROC/Ps since their 
inception in 1963, then provides both enrollment and fiscal figures for the five 
ROCs and nineteen ROPs that were in operation during 1969-1970. While these 
data are broken down by both ROC/P site and course subject area, two major 
components necessary for a cost-outcome anal3rsis are excluded- First, there is no 
information on student placement rates (or other measures of success other than 
program completion). Second, and more importantly, no attempt was made at 
relating the costs of operation to any of the measures that were obtained As such, 
that report reflects only a detailed listing of the status of California's ROC/Ps, not 
an investigation into costs related to outcomes. One important point surfaced, 
however, that may have a bearing on future investigations if it is still occurring 
today: 

The financial constraints upon school districts, coupled with the desire 
and need to expand vocational education opportunities for youth and 
adults, have combined to produce a number of instances where school 
districts acUvefy seek the financial resounds provided by Chi^ter 14 
[the ROC/P entitlement legislation] to support programs and services 
that have traditionally been financed through other means, and thus, 
in a number of instances, Chapter 14 financial resources have been 
construed to be "general aid" provisions for the support of all or most 
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of vocational education programs and services, (p. 6) 

Other reports were produced under this legislative mandate, though they 
continued in the same trcjk as the first. By 1976, the Status Report (California 
State Department of Education, 1976) was reporting on programs in over 66 
ROC/P sites. The report now also included information on both student completer 
and follow-up rates for different programs of study, as well as the calculation of an 
average cost per unit of Average Daily Attendance (ADA). This first attempt at a 
cost-outcome comparison was somewhat lacking, as it aggregated Single District 
(SD), Joint Powers (JP), and County Operated (CO) together calculating a single 
value only for each of the three kinds. Nonetheless, this was the beginning of a 
movement that was to continue into the next decade. 

It was little more than a year after that report was published that a decision 
was made at the state level to intensify the collection and analysis of information 
regarding ROC/P students. Van Zant (1978) began serious data collection with a 
statewide program to implement the "education program monitoring matrix", a 
method whereby standard reports could be used to collect and disseminate 
information on a number of program factors, including: program length, enrollment, 
completions, placements, and costs, including costs per enrolhnent, per completion, 
and per placement. This work also advocated the calculation of a benefit/cost ratio 
based on a societal benefit framework. The calculation methodology used for this 
benefit/cost ratio is extremely simplistic (it assumed that all unskilled workers are 
paid at the standard minimum wage and computes the benefit as the difference 
between the average wage of placed students and the minimum wage). Though 
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not rigorous, it did provide a bird's eye view of how cost and outcome data could 
be collected and analyzed from a California vocational education program. 

This led to a four year study of the effects of vocational education. 
Ostensibly performed to comply with part of the regulations in the federal Carl D. 
Perkins Vocational Education Act (PL 94-482) these studies sought to determine 
the effects of vocational education in both high schools and regional occupational 
centers and programs (Neasham, 1980, 1981a, 1981b; California State Department 
of Education, 1982a; Wright and Kim, 1982, 1983). Concerned primarily with 
federal requirements, these studies investigated program "effectiveness** only. The 
most recent of these studies obtained administrator feedback across nine different 
areas: quality, availability, student services, facilities and equipment, teacher-pupil 
ratio, teacher qualifications, student achievement, sex fairness, and services for 
special populations. Comparisons were made between high school programs and 
ROC/P programs (in three of the four studies), as well as internally between the 
nine areas of mterest. Differences in the ways in which these studies were 
conducted over the four years, the fact that the data were mostly self-reports from 
program staff, as well as the fact that no cost information was collected, make 
them only marginally useful for cost-outcome analysis. 

At the same time these studies were in progress, the State Department of 
Education, to comply with another legislative mandate (Cal Ed Code. § 52332 
(1982)) undertook several studies to determine the costs of ROC/P operations at 
a number of different sites (California State Department of Education, 1983b, 1986). 
The first of these studies investigated a stratified random sample of 100 courses 
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from which "information [derived from this sample] should fairly represent the 

costs of courses enrolling most ROC/P students" (p. 2). Following existing state 

mandated fiscal recording and reporting procedures allowed the study to coU.ect 

similar data in most respects from each of the participating sites. Where this 

methodology fell short was in the appropriation of indirect costs. 

While most of the programs show an excess of expenditures over 
revenue* it must be kept in mind that the calculation of "direct 
support costs" and "indirect costs" are normally the expenditures which 
create these imbalances. In the case of ROPs, these are very similar 
to a fixed cost which would occur regardless of whether the program 
was ofiered or not. . . . Procedures for establishing charges for 
administration between participating districts and the fiscal agent are 
varied and range from fixed percentages to in-kind contributions, (p. 
13) 

The latter report changed the data collection methodology, bowing to 
responses that the initial report was too fragmented to produce usable information. 
The second work focused instead on all of the costs at three ROC/P sites. Using 
much the same da^a collection strategy as before, this work showed both total 
revenues and expenditures, the total ADA, and the revenues and expenditures per 
ADA. While the first report showed an excess of expenditures to revenues the 
second report found the exact opposite. Like the first report, which only studied 
100 courses statewide, the second report suffered from a lack of generalization by 
iiivestigating only three sites (out of sixty-eight in operation during the year in 
question). Since only costs were considered (a characteristic of both works), little 
is fotmd that could be applied to cost-outcomes analyses. 

If this were not enough, a third movement was undertaken during the early 
1980's to collect information on vocational education students. Known generally as 



Qtial & Eff of CA R0C/P8 



Page 47 



CERC @ UCR, 6/9/89 



the Follow-up of Students and Employers (FUSE) or the Project for Student 
Employment Experiences (SEE) this multi-year investigation surveyed both high 
school and ROC/P students about their current work experiences. The major 
findings firom these years of work were that mor^ students were working in their 
third year out of school than in their first, that students working were employed 
in the field for which they had received training, and that they were also generally 
earning more money than those working in an unrelated field (Kim. and Wright, 
1984). As in prior studies, no figures for the cost of educating the students were 
collected, and the results are somewhat questionable since they were based on a 
small sample group with an uneven distribution. 

A similar effort was undertaken in the community college system, called the 
Statewide Student Follow-up System (SSFS). Fariand, Anderson, and Boakes (1987) 
describe a much more expensive survey effort relating students* stated goals in 
entering vocational education with the employment and educational experiences 
following their community college training. This study separated part time from 
fiill time employment and students who were seeking employment from those not 
seeking employment. This study also collected and analyzed the relationship 
between student's incoming educational goals to their subsequent outgoing 
education and employment status. It also collected student comments pertaining 
to educational and subsequent employment experiences. This effort also did not 
address the costs issue, but did take significant strides in clarifying outgoing 
student status. For the first time, performance status was also related to incoming 
goals across a series of different educational objectives. In this way students who 
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had changed objectives during training, as well as those only seeking training for 
self-improvement or field investigation, were identified separately from those 
seeking vocational training for eventual placement in the field. 

Other studies have evaluated the effectiveness of vocational education from 
the employe's point of view, rather than from the students or scl^ools. ^ilms 
(1983) queried downtown Los Angeles and Torrance employers to uncover what 
traits they desired in a new employee. Analyzing data gathered from numerous 
interviews, they determined that employers placed the greatest value on new 
employees who were hard working and followed the employer's rules over those 
who simply had advanced skills training. He suggests that this is because 
employers believe new employees must at least come to work regularly, punctually, 
and do what is expected of them before they can be engaged in higher skills tasks. 
Technological advancements and special skills were only important in about one- 
quarter of the cases. If this is true, then vocational education should be 
concentrating more on general skills and employee attitudes than on the 
development of specialized skills acquisition, leaving such advanced training to the 
purview of the eir.ployer. 

Carvell Education Management Planning (1984) brought the entire issue of 

outcome evaluation into focus with a lengthy and complete treatment of the 

concept of placement rate evaluation. 

For the purposei» of the study, the placement rate concept was defined 
as the number of students who received training and are available for 
work after training, divided into the number of students who find 
work. (p. 2) 



Qual & Eff of CA ROC/Ps 



Page 49 



CERC @ VCR, 6/9/89 



This definition, still in use today by the California State Depiirtment of 

Education (California State Department of Education, 1988a), limits the evaluation 

of successful outcomes only to those students who obtained employment. Carvell's 

report criticizes this limitation, stating that the seven outcome categories defined 

hy the state (related employment, unrelated employment, related 

education/advanced training, unrelated education/advanced training, military 

service, not in the labor force/not seeking work, and unemployed/seeking 

employment) did not allow for students enrolled in vocational courses for the 

purposes of upgrading their skills. Such students, called "Career Improvers" by 

CarveU, go unrecognized or misreported under the current sjnstem. The variety of 

findings lead them to recommend that 

. . . standardized definitions for each postprogram placenaent option, 
program, completer, and leaver must be agreed upon and adopted to 
carry out student foUow-up studies, data reporting, and analysis of 
program outcomes, (p. 3) 

Also unknown by most vocational education administrators is the starting 
emplojnnent status of students at the time they enter a program of study as well 
as their intentions (or goals) for taking such a program. Without these components 
little can be said about the true impact of vocational education. 

These issues are perhaps best addressed in a comprehensive stuc^ done by 
Stem, Hoachlander, Choy, and Benson (1986). Their findings, controversial to this 
day, show that vocational education students from comprehensive high schools aiid 
ROC/Ps (combined) experience a higher unemployment rate than the general 16 
to 19 year old population (26% as opposed to 23%). They also determined that 
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vocational education was not, as a rule, an effective means of altering student 

dropout patterns. One interesting point they make in regard to regional 

occupational centers and programs relates to the voluntary' status of students: 

The essence of the ROC/ROP program, however, is not to be found 
in its statewide administrative structure but in the fact that 
enrollment of students in ROC/ROP programs is strictly voluntary. 
Funding by the state - standard practice - is based on attendance, 
which is closely related to enrollment. Hence, the size of a given 
ROC/ROP's budget, from which many good things flow, is a direct 
function of its ability to attract students. This gives an 
entrepreneurial cast to ROC/ROP administration. Somehow, the 
director and his or her faculty must convince an appropriate number 
of students (better yet, a growing number of students) that enrollment 
in their ROC/ROP is Vorthwhile " (p. 2). 

Several methodological problems may make these findings less powerful than 
they at first appear. It is possible that the students who would enroll in vocational 
education courses would be less likely to obtain employment than the regular 
population. If this is the case, and there is no way to tell from their data, the 
efforts of vocational education might be assisting this group by making them more 
like the regular population (even though they do not exceed it). Nonetheless, the 
authors make several recommendations regarding how vocational education can be 
restructured to shift the training for specific skills to ROC/Ps. High schools would 
then be free to concentrate more on general education and what they refer to as 
"enterprise training" (combining production with education, including all students 
in some form of vocational education, teaching teamwork, integrating vocational and 
academic education, and encouraging active mquiry). Further work appears needed 
if we are to learn whether it is vocational training, academic education, some 
combination of the two, and/or factors outside of the school that determine the 
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true correlates to post-high school employability and employment success 

Conclusions from Prior Research 

Tie literature makes a number of important joints relative to the 
applicat on of cost-effects analysis to vocational education. First, there is a 
sufficient number of works dealing with the theoretical division and applications of 
cost strategies lo outcome indices. These studies generally follow a separation into 
four different kinds of studies: cost-feasibility, cost-utility, cost-benefits, and cost- 
effectiveness. Each method has been described and individual strengths and 
weaknesses highlighted While numerous studies have been done applying these 
theories to other fields of inq»:iry, far fewer have been accomplished in the realm 
of education. When narrowed to vocational education, and specifically regional 
occupational centers and programs in the state of California, only a handful of 
works a^jply. 

Readings of cost-outcome analyses across inter st areas have shown that, 
often times, the studies are fU conceived and poorly executed. Program cost 
components are usually well identified but are not typically tied to a theory of cost 
analysis, leaving the reader to surmise the utility and application of the worhing 
firework. Cost-outcome theoreticians make a point of stating that such a 
framework must be acknowledged up front in order to drive the cost data 
identification and toUection, not the other way around. Less well defined are the 
components of programmatic outcome. Outcomes intrinsic to the program are 
usually well understood but are felt to be of little applicability outside of the 
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pregrtim (such as the number of completers of a given course of study). Attempts 
at measuring extrinsic outcomes often bad to questionable conclusions since, while 
statistical relationships can be formed between the program components and the 
measured outcomes, it is usually not clear whether the outcomes being mea? ured 
are truly a result of the program experiences as opposed tc those that might have 
occurred naturally without the intervention (i.e., inadequate control variables or 
non*treatment cohort groups). 

Previous studies in vocational education have tended to avoid this problem 
by only addressing either the costs or the outcomes of a program. While 
interesting in themselves, these studies do not further the knowledge of cost- 
outcome analysis within this field of education. California's ROC/Ps are just one 
example of a program being assailed with calls for accountability in both areas of 
concern. Yet it is this very call for cost and outcome accountability that requires 
an integrated investigation. 

Operationalization of this Study 

In order to conduct the research described above the various components of 
cost and outcome data would need tc be collected about selected courses and 
environmental data collected about selected sites. An examination of available 
records showed only some of this data to be currently available or in press. 
Finandai data, both revenues and expenditures, about all of the ROC/Ps was 
available from the California State Department of Education. Unfortunately, this 
data was aggregated to the site level and could not be disaggregated accurately on 
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a course by course basis. A portion of the student oi'tcome information was 
avaOable on a course by course basis from the State, but was generally not 
available until almost a year following course completion. Some of the 
enviromwental data was avaibble (California State Department of Education, 
1983c), but some did not exist ai?ywhere except at the individual sites. It had 
become apparent that, it order to get the detail of data required for this analysis, 
the information would need to be requested from the ROC/Ps sites themselves. 

State administrative regulations required that each ROC/P conform to 
standard accoupting categories when reporting their fmancial information on annual 
reports (J-300-ROP, for example). Although this data was not available course by 
course, examination of these categories (Table 6) did provide an indication of how 
the ROC/Ps managed their financial data. The questioned remained of whether 
this information was available within each site on a course by course basis. 
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Description 



Soiirce/Obiect Code EDP Number 



Revenues 

Revenue limit Sources 8010-8099 200 

Federal Revenues 8100-8299 280 

Other State Revenues 8300-8599 450 

Other Local Revenues 8600-8799 697 

Piior Year Revenue Adjustments 8800-8899 001 

Total Revenues 699 



Escpenditures 

Certificated Salaries 

Classified Salaries 

Employee Benefits 

Books and Supplies 

Services, Other Operating Expenses 

Capital Outlay 

Other Outgo 

Direct Support/Indirect Costs 
Prior Year Expenditure Adjustments 
Total Expenditures 



Table 6: J»300-ROP Accounting Categories 



Student outcome information from the State Department of Education 
detailed, for each course conducted during 1987-1988, the number of student 
enrolled in that course as well as the number of gtudent who completed the course 
(California State Department of Education, 1988a) . This information was further 
subdivided according to the numbe* of students who were concurrent students 
(attendmg the ROC/P while attending a regular high school) versus those who 
were nonconcurrent students (those attending the ROC/P but not enrolled also 
enrolled in a high school program). This distinction could prove important, 
especially due to the changes made during the past few years encouraging more 
high school students in ROC/P program (see the section on ROC/P history for 
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1000-1999 627 

2000-2999 651 

3000-3999 702 

4000-4999 729 

5000-5999 762 

6000-6599 780 

7100-7299 846 

7300-7399 855 

7400-7599 002 



more on this topic). 

Placement information was mandated from the sites only for students who 
completed tiieir course of study. Those students still enrolled in the ROC/P at the 
end of the reporting period, as well as those who had dropped out of the program 
prior to completion, were not required to be included in the follow-up process. 
This is not to say that students who left the ROC/P program prematurely did not 
seek or obtain emploj^ment. Indeed, it is possible that some did. Students who 
obtained employment in the field in which they were being trained but who had 
dropped out of the program early could be included as part of the sample statistics 
if information on these students had been collected and retained at the site level. 

Placement information that was reported to the state adhered to the seven 
foUow-up categories mandated for annual reporting (California State Department 
of Education, 1988a): 

Status Unknown 

Persons failing to return the follow-up survey or giving 
incomplete or multiple responses. 

Military Service 

Persons reporting full-time military service. 

Pursuing Additional Education 

Persons pursuing additional education even if they are 
employed (but only in a field NOT related to their 
vocational training), NOT in the labor force, or 
unemployed The additional education must be in 
academics or advanced vocational training. 

Other Reasons 

Persons who are deceased; or persons not employed AND 
not pursuing additional education. 
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Employed - Nonrelated Field 

Persons employed in a job not related to training AND 
not pursuing additional education. 

Employed in Field Trained 

Persons employed in a field related to training although 
they may be pursuing additional education. 

Unemployed - Seeking Employment 

Persons who are not emj/loyed AND not pursuing 
additional education BUT who are seeking employment. 

Table 7: VE-80-C Follow-up Categories 

Close examination of these categories reveals that they are not necessarily 
complete nor mutually exclusive. For example, it would be possible for a student 
to have obtained employment (full or part time) while also pursuing additional 
education. Since the instructions provided from the state do not address this (and 
similar) problems, and since it is possible that such problems could arise, it would 
be necessary to determine how the different ROC/Ps sites resolved the problem(s) 
in categorizing their completed students and filling out the VE-80-C form. 

Finally, not all of the environmental information could be obtained from 
existing records. For example, only the local sites could accurately describe the 
population characteristics of their respective service areas, since exact definitions 
of the boundaries of these service areas do not exist except at the sites themselves. 

Before data could be solicited from the sites these questions had to be 
resolved in order to insure that data collection would proceed smoothly and 
miiformly. If course level financial data were not available from the sites, or there 
were extreme differences between sites in terms of their handling of overlapping 
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outcome categorizations, the unit of analysis and basic methodology would have to 
be reconsidered. Therefore, a series of short visits were made to twelve of the 
seventy sites for the purpose of clarifying these issues prior to the commencement 
of Ml fledged data collection. 

Site Visits 

The protocol presented in the Appendices (Appendix A) was utilized in 
twelve site visits for the purpose of clarifying these issues of data definition and 
availability. These sites visits were conducted over a three month period from 
October through December, 1988. The actual sites were chosen to represent the 
three different organizational types of ROC/Ps (Singlt District, Joint Powers, and 
County Operated). A visits involved a maximum of one day including round trip 
travel time, with a maximum of four hours on site. 

The chosen site's Directors were initially contacted over the telephone to 
solicit their participation in the site visit. A brief overview of the visit was 
provided to each, along with a description of the visit protocol. All sites contacted 
agreed to participate, with a date and time for the visit tentatively agreed upon. 
Following the telephone conversation a confirming letter was mailed to each site 
(^pendix B), giving more detail regarding the purpose of the visit, a recap of the 
telephone conversation, and identifying the ROC/P stafi" the Director had agreed 
to have participate in the visit. It had been requested of each site that the 
Director, the Business Manager/Accountant, and the 
Attendance/Enrollment/Student Records Manager attend the meeting. A few days 
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prior to the visit a reminder telephone call was pkiccci to each Director. 

The number of ROC/P participants at the visits ranged from two to eleven, 
alw£0^s including at least the Business Manager and Student F^cords Manager. 
Except for one site the ROC/P Director also attended and participated in the 
meeting (in the case of the one site the Director was called away to an emergency 
meeting shortly after the site visit began). Time on site ranged from one and one- 
half hours to three hours. 

The site protoool was followed in a loosely-structured interview format. All 
sites agreed with the State*s definition of concurrent and nonconcurrent students, 
as well as with the different outcome statuses. Of the twelve sites, seven defined 
a student "completer" entirely on the basis of satisfactory academic achievement 
and one site defined a completer as "whether or not the student got a job". The 
remaining four used a mixture of satisfactory academics ard employment 
attainment. None the less, each site clarified that a student was classified as a 
completer only if they could demonstrate, through some combination of academics 
and/or active employment, that the student had obtained sufficient job skills in 
then* program of traimng in order to be employable in that field 

All of the sites indicated that the enrollment and completion data on all of 
their courses had already been submitted to the State Department of Education. 
Each felt that the reported data was complete and accurate. Only one site had 
already submitted student follow-up placement data. 

When queried about their follow-up strategies three sites reported to 
following-up on both completers and leavers, with nine sites performing follow-up 
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studies on the completers only (as required by the State). Actual strategies for 
performing the follow-up studies were quite vmed, including: teacher initiated 
contacts, telephone surveys, mail surveys, and special promotional events. Each 
ate followed their own timetable for the follow-up process depending on the 
methods being used Nine of the sites foUowed-up on all of the completers in all 
of the courses held during the year, two sites sampled all of the completers (and 
leavers, in one case) but only from a few selected courses, while the single 
remaining site took a partial sampling of completers from all of the courses held. 

All twwive sites were able to produce relevant site-level Hscal documents and 
to describe their particular method for financial record keeping and transacting. 
Nine of the sites maintained fiscal information on a course-by-cuurse level, with 
seven of these able to also report fiscal information on a section-by-section or 
district-by-district (for joint powers agreement p^'^ county operated sites) basis. 
The three sites that did not maintain fiscal datt way uniformly reported that 
fiscal information could only be kept on the site It ei ^since "some courses make 
money and some loose, but overall it has to balance") and that it would be difficult 
but not impossible to disaggregate their fiscal data to a course-by-course basis. 

Participants were given an opportunity to^ express mdividual opinions and 
ideas not part of the protocol prior to the end of the visit. Following each visit a 
letter of appreciation was written to each site thanking them for their input and 
informatiou (Appendix C). 

Two Kinds of Surveys 
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It had become apparent through the twelve site visits that there were two 
classes of information available at each ROC/P site. The first was that which was 
course spedfic, such as: number of students enrolled in, completing, and leaving a 
particular course, the costs associated with the course, and the income generated 
by that course. Certain factora, however, were fairly universal for all courses at 
a particular site. These mcluded: the way in which the site determined which 
students were course completers and which were leavers, the methodology used for 
the foUow-up study, and how certain costs were determined (such as contractually 
negotiated teacher's salaries). A single survey instrument asking for data in both 
areas across multiple courses at a single site would encounter numerous repetitions 
on the site-specific information; thus, it was decided to employ the use of two 
survey instruments. The first would ask for data specific to selected individual 
courses while the second would request information about site-specific information. 
It was expected that, depending on the number of courses to be included m the 
survey, an individual site would receive only one of the site-level surveys and, 
probably, more than one of the course-level surveys. 

The course-level survey instrument was developed first. Since an ind idual 
site might be expected to complete several of these an emphaas was placed on 
keeping this instrument both short and relatively easy to complete. The siurvey 
was divided into two parts: one to collect the "output" data (student follow-up) with 
the other to collect course "input" information (course costs and expenses). The 
layout of the items generally followed the standard format for information 
reporting ahready administered by the State of California, with modifications for 
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dariflcation made necessary by the insights gained through the site visits. It was 
expected that sites which only collected and reported information according to the 
state mandates would be able to complete the survey quite easily, while those who 
collected additional information would be able to include that infom ation with only 
m i n ima l additional work. Space was left on each survey for the addition of a self- 
stick label containing the specific course information (course identification number 
and title). This instrument is reproduced in Appendix D. 

The site-level survey was developed next to collect the information specific 
to that site but not subdividable on a course by course basis. The sections on 
student follow-up and finances were repeated, but with a focus on the site as a 
whole instead of on individual courses. Questions were included on the first page 
to collect some of the environmental data (such as the site's service area 
population size, density, and local economic status). Additionally, several open 
ended questions were added to the last portion of the instrument to determine the 
extent to which volunteerism, donations, and less than market value purchasing 
was occurring at the site. This instrument is reproduced in Appendix E. 

Selection of the Sample 

The VE-80-B database for 1987-1988 was obtained from the California State 
Department of Education (1988a). This computerized data file contained the total 
enrollment and completion figures from all of the individual course offerings fircm 
each of the State's seventy ROC/P sites, a total of 2,958 different courses. This 
data represented the total of all courses available for investigation during the study 
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year. Preliminary analysis showed that the diiJtribiition of courses between sites 
was far from uniform. One site had as few as six course titles offered while 
another as many as 274. Distribution by subject area Hkewise ranged from CBEDS 
subject classifications which had only a single course offered in them to ones which 
had as many as 177 different titles. The numbers of students enrolled into these 
courses also varied considerably, from as few a eleven students taking coiurse work 
in a single CBEDS subject area to as many as 22,943 in another. 

For the studies findings to be as generalizable as possible the sample 
selected needed io be representative of the scope and nature of ROC/P courses 
being conducted across the state. Since only seventy sites existed it did not 
present an undo burden for each to be included in the research, although it did 
not seem necessary that all 2,958 courses be included. Therefore, a strategy was 
undertaken to determine which course titles were the most frequently occurring. 
A selection could be made from amongst these. 

The data were organized according to CBEDS subject-area classification code 
number and were then cross-tabulated by ROC/P site in order to show the 
numbers of courses and students enrolled in these courses within each CBEDS 
subject area. While most of the sites conducted at least one course in each of the 
more popular subject areas this was by no means uniform across the state. 
Furthermore, the CBEDS classification system tends to group like m^or subject 
areas together (such as number 4000, Agricultural Production, and number 4002, 
Agricultural Mechanics) while not addressing similar groupings in the minor areas 
(such as between number 4002, Agricultural Mechanics, and number 4803. Auto 
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Mechanics). 

As this strategy did not produce a clean separation of courses " v either 
common subject areas or student enrollments the courses were further aggregated 
into broader subject area domains than employed by the CBEDS. Courses that 
were offered at mor j than one-half of the seventy sitrs and to at least one percent 
of the total statewide student populatiun were chosen. This resulted in the 
selection of thirty CBEDS numbers accounting for 2,080 of the total titles 
(approximately 70% of the statewide total), accounting for 266,425 enrollees (74%). 
These thirty CBEDS numbers were further reduced according to course subject 
content to nine different subject area domains, eight of specific content areas and 
the last including all other unclassified codes. Each of the seventy ROC/P's course 
offerings was then separated, according to CBEDS number, into the different nine 
classifications. The distribution of all of the courses offered statewide into these 
nine domain classifications is depicted graphically in Figure 4, and the student 
enrollment statewide is depicted in Figure 5. 
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Regional Occupational Center/Program 

Statewide Course Titles by Domain 



Other 55% 




Regional Occupational Center/Program 

statewide Course Enrollment by Domain 




Using only the eight specific content area domains a random selection was 
made of a single course title from each domain from each site. Since rot every 
site offered at least one course in each of the eight domains (some of the smaller 
ROC/Ps only ofTering a handful of courses in very specific content areas) not every 
site was able to have a total of eight courses selected. The selection did, however, 
result in 482 courses being selected, an average of 6.88 courses per site. Table 8 
presents the eight subject area domains, the total number of courses selected in 
each domain, and the number of students enrolled into those courses. 



Subject Domain 
Auto Repair 
Office Occupations 
Data Processing 
Retailing 

Health Occupations 
Quantity Food 
O)smetology 
Construction 

Total 

Statewide Total 
Percent of Statewide 



# and % of Courses 

64 13.9% 

65 13.5% 
63 13.1% 
61 12.7% 
69 12,2% 
61 12.7% 
47 9.8% 
59 12.2% 



482 
2958 



100.0% 



16.3% 



# and % of Enrolles 

13,655 12.6% 

26,690 24.6% 

21,871 20.1% 

13,200 12.1% 

6,960 6.4% 

9,680 8.9% 

10,374 9.5% 

6,329 5.8% 

108,759 100.0% 
358,318 

30.4% 



Table 8: Sample Distribution by Subject Area Domain 



This selection produced a distribution of courses by ROC/P site with only 
one site having just two courses selected, one site with three courses, one with 
four courses, twelve with six courses, and the remainder with either seven or eight 
courses selected. Examination further revealed that those sites which had fewer 
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than seven courses selected were those sites in which the total number of courses 
ofTered at the site was also smaller. Thus, this sampling^ strategy allowed for 
participation of all of the ROC/P sites in the course-level survey process in such 
as way as to allow for a reasonable number of surveys from each site as well as 
the comparability of multiple sites across different subject area domains. 

A survey packet consisting of a cover letter (Appendix F), a site-level survey, 
from two to eight selected course-level surveys, and a postage-paid return envelope 
was prepared for each of the seventy sites. The packets were mailed to the sites 
on January 27, 1989. Subsequent telephone contact with the sites determined that 
three of the seventy sites could not acknowledge receiving their packets and one 
8ite*s packet was returned with an undeliverable address, so replacements were 
sent out to those four sites during the middle of February. Sites which had not 
responded by the end of February were contacted by telephone in an effort to 
encourage their response. This procedure was repeated in mid-March, with the 
data collection finally closing at the end of the first week in April, 1989. 

Analysis of the Data 

Fifty-one of the seventy sites (73%) returned their survey packets. Of these 
six sites had to be disqualified due to a large volume of incomplete responses, 
leaving a usable sample of 45 sites (64%). These remaining sites accounted for 307 
differ I It course-level surveys, or «>4% of the total of 482 sent out The total 

m 

impacted student enrollment for the usable sample was 70,279, or 65% of the 
108,759 students included. 
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Surveyed 



Obtained 



70 



Totui Sites 
Valid Sites 



51 (73%) 
45 (64%) 



482 



Total Courses 
Vaiid Courses 



347 (72%) 
307 (64%) 



108,759 



Total Enrollment 
Valid Enrollment 



(82%) 
70,279 (66%) 



Table 9: Survey Response Distributions 



Analysis of the data proceeded in three steps. The first analysis was to 
determine the adequacy of the returned sample. It was found that a representative 
sample of the statewide ROC/P population had been obtained. The survey 
responses were then examined through frequency and correlational analyses for the 
purpose of data clarification. Cost-effect ratios were calculated for each site on 
each of the outcome measures for each course across the eight subject area 
domains. Analysis of variance statistics showed bignificant differences in certain of 
these ratios, with multiple comparison techniques localizing these differences to 
specific subject area domains and outcome measures. 

Adequacy the Sample 

The first consideration was whether the usable sample was similar in 
distributionai characteristics to t)ie total population. Since the exact parameters 
of the distribution were expected to vary depending on sub-group organization (such 
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as ROC/P location or organizational t>'pe) a distribution-free (non-parametric) 
statistic was used for this test. A Kolmogorv-Smirov Z of .725 (p = .669) was 
obtained using enrollment as the test measure, determining no significant difference 
between the obtained sample and the surveyed sample. 

The next concern was if the response distribution was similar to the 
surveyed distribution according to both ROC/P location and oi^izational type. 
Chi-square tests were performed row-wise (by organizational type), column-wise (by 
location), and table-wise (organizational type oy location). The obtained sample was 
not significantly different from the surveyed Lample on any of these three measures 
(row-wise (x^ « 0.498, p ^ .780), column-wise (x^ = M16, p « .761, and table- 
wise (x^ = 2.788, p « .972). The frequency chart is presented in Table 10. 
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Central Coastal Northern Southern Row Total 



Surveyed 


7 


10 


14 


8 


39 




10.0% 


14.3% 


20.0% 


11.4% 


55.7% 


CO 














4 


7 


7 


5 


23 


Obtained 


8.9% 


15.6% 


15.6% 


11.1% 


51.1% 


Surveyed 


5 


7 


3 


11 


26 




7.1% 


10.0% 


4.3% 


15.7% 


37.1% 


JP 














4 


4 


1 


10 


19 


Obtained 


8.9% 


8.9% 


2.2% 


22.2% 


42.2% 


Surveyed 


1 






4 


5 




1.4% 






5.7% 


7.1% 


SD 




None 


None 








1 






2 


3 


Obtained 


2.2% 






4.4% 


6.7% 


Surveyed 


13 


17 


17 


23 


70 




18.6% 


24.3% 


24.3% 


32.9% 


100.0% 


Col Total 














9 


U 


8 


17 


45 


Obtained 


20.0% 


24.4% 


i? 1% 


37.8% 


100.0% 



Table 10: Frequency Chart by Organizational Type and Location 



A third area of sampling adequacy centered on the response rate among the 
eight subject-area domains. A frequency chart for the domains was formed 
comparing the number of courses surveyed in each area to the number of responses 
received (Table 11), A chi-square test showed no significant difference between the 
sample obtained and the sample mailed out (x^ = 1.148, p = .992). 
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Subiect Area Title 


Surveyed 
Numb«*r Percent 


Obtained 
Number Percent 


Auto RBDdir 


67 


13.9% 


42 


13.7% 


Office O^^mnti nnsa 


65 


13.5% 


39 


12.7% 




63 


13.1% 


39 


12.7% 


Retailing 


61 


12.7% 


42 


13.7% 


Health Occupations 


59 


12.2% 


38 


12.4% 


Quantity Food 


61 


12.7% 


37 


12.1% 


Cosmetology/Barbering 


47 


9.8% 


34 


11.1% 


Construction 


59 


12.2% ' 


36 


11.7% 


Total 


482 


100.0% 


307 


100.0% 



Table 11: Frequency Chart by Subject Area Domain 

These tests show that the obtained sample is not substantially different from 
the population in terms of distribution. Even though not all of the sites responded 
to the survey a sufficient number did respond from a cross section of the 
organizational types and geographic locations in order to make the sample worthy 
for thij analysis. In other words, there is no evidence that any one (or group) of 
ROC/P organizational types, locations, number of enrollees, or course domains 
re^nded more or less frequently than any other. Non-response, a potential 
problem in any survey data (Osborne, Rush, & Fondacaro, 1986; Qark & Finn, 
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1989, April; Green, 1989, April) appeal's to be a randomly distributed process 
unconnected to the above mentioned variables. 

Cost-Effect Ratios 

Further analyses focused on identifying commonalities among the different 
measures. Each of the eight subject areas share essentially tl; 3 same leaver rates. 
The rate of continuers is highest in Cosmetology courses (perhaps owing to their 
typically longer duration), with the remaining courses all sharing a common rate. 
Courses in the health occupations exhibit the highest rate of completions, although 
this is not significantly different from other subject areas except cosmetology and 
construction. This data is depicted graphically in Figure 6. 

Data on the eventual outcomes of course continuers was not reliably available 
from the individual sites, since multi-year student tracking is not conuno^y done 
for this purpose. If. however, course continuers are assumed to complete and leave 
in the same proportions as the rest of the students in their program, most of these 
differences in the completer and leaver rates disappear. These revised differences 
are reported in Figure 7. 
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Figure 6: Outcome Status by Three Program Areas 

Regional Occupational Center/Program 

Outcome Status by Three Program Areas 




Program Area 



Continuers 



Leavers 



Completers 
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Figure 7: Outcome Status by Two Program Areas 

Regional Occupational Center/Program 

Outcome Status by Two Program Areas 

Completers and Leavers Only 
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Most of the sites reported conducting follo^v-up studies on completed students 
only. A few sites reported following-up on all of their students, although this 
resulted in too few responses for reliable statistical analysis. From the availabk 
follow-up data two measures of student outcome achievement were calculated. The 
first represents general job placement, and follows the methodology used by the 
VE-80 reporting system. The VE-80 definition creates a ratio of the number of 
students who have secured employment, whether in the field in which they were 
trained or another field, over the total number of students who were seeking 
employment. Employment in the military is not considered for this ratio. The 
second measure is a more general one representing overall student positive 
outcomes. This ratio follows the mission statement .for ROC/Ps as outlined m the 
Education Code. It was calculated by determining the total number of students 
who were either emplo^^d in any field, who enlisted in the miUtaiy, or who went 
on for additional education as a ratio of all students who went through the 
program. A comparison between the completer rate, VE-80 definition follow-up 
rate, and mission statement definition follow-up rate is presented in Figure 8. 

In both cases the ratios were calculated using only figures available from 
contacted students, ignoring the site's non-respondents. This procedure is 
acceptable if the samples obtained by each site are representative of all of the 
studen:« in the course. Unfortunately, there was no way to determine in this 
study whe her the non-respondents were different from those who did respond, 
either positively or negatively; thus, the obtained responses were used as 
representing the entire student body with appropriate cautions on this regard. 
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I Regional Occupational Center /Program 

5 Percent of Students in Outcome Status 




§ Completers M VE-80 Def fZ] Mission Stmt 
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On the fiscal side, mo:?t sites rej)orted revenue limit sources as their major 
or sole source of funding for individual courses. Minor exceptions were evident 
though it is clear that ADA reimbursement is the ROC/P's primary method of 
revenue generation. Expenditures tended to divide along two basic lines. Teacher 
salaries and benefits constituted the larTest component, with contracted services 
and support charges (from participating districts and private schools) running a 
close second. The fiscal data are summarized graphically on the following pages. 

The distinction between ROC/P conducted courses and those contracted to 
school districts or private schools showed clearly in the method of reporting. 
Courses that were conducted by the ROC/P itself provided many lines of detailed 
information while those courses that were contracted out to a district or third- 
party school tended to have fewer lines of expenditure data, typically only a single 
reporting of the total contractual amount. This was expected, since in the 
negotiations for these courses the details of specific expenditures are largely in the 
hands of the district or private school While identifiable, this con^'^unds the 
creation of a single cost-effects ratio, since it is clear from the site visits that 
ROC/Ps approach fiscal negotiations with participating school districts quite 
differently than when dealing with outside vendors. 
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Regional Occupational Center/Program 

Average Division of Direct Revenues 
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Regional Occupational Center/Program 

Average Division of Direct Expenditures 



CertiticiilHi I .11;, 




For most courses, direct cojjts operation are between $100 and $200 per 
enrollee below the total income. On average, direct costs for courses in quantity 
food exceed income, while in cosmetology revenues exceed per course expenditures. 
ROC/P indirect costs (central office staffing, benefits, facilities, and maintenance) 
can account for 15% or more of the overall budget. In order for programs to 
operate with the little revenue margin in most of the program areas courses like 
cosmetology and retailing must exist. It appears that without courses like these 
it is questionable whether ROC/Ps would have suHlcient excess revenues for 
indirect cost operations. The distribution of these direct and indirect costs is 
presented below and in Figure U. 



Subiect Area 


Direct Costs 


Indirect Load 


Total Costs 


Quantity Food 


$2,496 


-$202 


$2430 


Construction 


$2,411 


-$8 


$2438 


Health Occup 


$1867 


$526 


S2400 


Auto Repair 


$1850 


$556 


$2388 


Data Process 


$1693 


$544 


$2328 


Office Occup 


$1693 


$614 


$2343 


Retail 


$1516 


$798 


$2371 


Cosmetology 


$1094 


$12^3 


$2323 



Table 12: Average Cost Distribution per ADA 
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The next step of the data analycii;* involvLMl the coinpiitaiion at' the actual 
cost-effect ratiovS. Three differonl seL^ uf ratios were computed comparing each of 
the eight courses. Each ratio consists of t!ie total direct expenses incurred to 
operate each of the eight courses, expressed on a per ADA basis, divided by the 
number of students calculated (on the above percentages) to be in ihtt outcome 
category. While these numbers are not useful for determining the cost of providing 
each kind of outcome on a per stutlcnt basis (because of the calculati3n factors 
previously stated), they are important for inter-course comparisons- In genera^ the 
lower the number the less is costs to create each student in that outcome category 
for that course of instruction. This information is summarized in Table 13. 





Completer Job Plac 


Migsion 


Auto Repair 


63 


63 


69 


OfBce Occupations 


40 


45 


45 


Data Processing 


62 


60 


77 


Retailing 


61 


130 


135 


Health Occupations 


46 


52 


52 


Quantity Food 


132 


143 


152 


Cosmetology/Barbering 


46 


57 


56 


Construction 


148 


153 


164 



Table 13: Co.st-Effect Ratios by Course Area 



One way analysis of variance tests determined that differences exist between 
the ratios for only the completer column (F = 3 6776, p = .0008), with no 
difierences apparent in the remaining three measures. A Student-Newman-Keuls 
multiple range test was applied to the data of the completer column to focuo the 
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differences discovered through the AN()\'A, This test determined that construction 
courses were significantly more costly per a>ini)leter all oth':?r courses except those 
in quantity food. Courses in quantity footl were also found to be more costly per 
completer than courses in office occupations or auto repair. The remaining 
comparisons were not significantly different from each other e%'cn though there 
appears to be a large range in the average ratios. This is due to the v^-ide variation 
in both costs and outcomes rates for these two measures. 

A dose examination of this data reveals a possible explanation for this 
finding. On the whole, the average rates for leaving a course, securing 
employment, or obtaining any desirable outcome are essentially constant across all 
of the eight subject areas. Variations of up to twenty-five percentage points are 
present in some cases, but the averages tend to be very nearly the sanie. Only the 
rates of course completion differ significantly between courses, ranging from m 
average low of 37.32'^ for courses in Cosmetology to a high of 64.12% for courses 
in the health occupations. Variations are also evident in course costs, though they 
generally follow the pattern already mentioned with the exception that courses in 
Cosmetology are uniformly the least expensive per ADA of the eight subjects 
studied. It is only the matching relatively low completion rate in Cosmetology' that 
does not allow it to be shown as a more cost-effective course for producing 
completers. 

This data suggests that there is not a clean match between course 
expen,.litures and course outcomes, at least across three of the four outcome 
measures and most of the course subject areas. Yet there are variations to a 
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significant degree in both tiie course costs and, to a sJomewhat let^scr degree, in the 
outcomes produced. These outcomes would be better pixnlieted by variables other 
than course expenditures, 

StructunU equation analysis, in the form of patn analysis, was undertaken 
to determine the relationship between actual expenditures and the three different 
outcome measures. Results from these tests, though statistically significant, were 
not sufTicientiy strong to enable their im-lusion in this report. The data reveals 
that a mild contribution (approximately one-quarter of the explainable variation) 
is due to local population socio-economic factors. These include: ROC/P service 
area geographic size, population size, population denisity, and median family income. 
The more rural, less densely populated, largtr geographic areas in a more afHuent 
area seem to have a slight advantage in producing the measure outcomes. The 
bulk of the explainable variation, however, seems to be attributable to factors 
under the control of the ROC/P but unrelated to either site-level oi course-level 
financing. Whil increased fmancing of ROC/Ps will allow for service to more 
students in more courses, fniancing does not influence the rates of outcome success 
for the courses measured. Other factors, such as course topic, numbers of sections 
and locations offered, and total enrollment (related to class sizes) are related. 
Although not the focus of this research, these findings suggest that well managed 
ROC/rs - ones in touch wi h the desires of their students and the local work 
community - do achieve better rates of oiUcomcs. Further research is needed on 
this question before firm conclusions could be drawn. 
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Management Information System 

Information for the planning of y niunaguniunt information system wast 
collected in two difTerent ways: 1) open-ended questions were included on the site- 
level survey to ascertain key points of data collections and utilization, and 2) a total 
of twenty-two sites statewide were visited, twelve for one-day visits and ten for 
two-day extended visits. 

Responses to the Open-Ended Questions 

In addition to obtaining site total data for student enrollment, follow-up, and 
financial information several open-ended questions were included as part o'' the 
survey. These questions were intended to obtain more detailed information about 
specific areas for which the quantitative numerical information was possibly 
insufficient. Two specific areas were targeted for further investigation, based on 
kiformation collected during the twelve one-day site visits. 

The one-day site visits had shown that different methods exist between the 
sites in terms of both how the dilTerent out-status categor js were defined and how 
the follow-up data were collected. As part of the site-level survey the ROC/Ps 
were asked to describe their follow-up process. Significant differences in terms of 
how the follow-up categories are defined, or how the data are collected, would 
affect the comparability of these data between sites as well as the usefulness of any 
potential findings. 

In a Hke manner questions arose during the one-day site visits about the 
financial data. The financial statistics being requested were mtended to represent 
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the typical operations of both the !?eiuutcd cotir-.ei? and the site, iievcnues or 
expenditures that were of a one time. r«\Hricted. or non-periodic nature may 
represent a special event that ocrurrcd during the year. The amounts, if they 
exist, may have to be treated separately in order to obtain a true understanding 
of the course or site under investigation. 

One of the basic premises of a cost-effects analysis is that there must be 
variation in either the costs or the effects (or both) in the program under study. 
It is also assumed that variations are due to choices made independently by the 
controlling organizations, with a high dejjree of uniformity in the regulations 
regarding the ways in v/hich these decisions are reached. If significant influences 
exist differentially among the sites observed variations in the cost and/or effects 
of the program may occur more as a result of these influences and less as a result 
of programmatic variations. To determine the possible extent of restrictions several 
open-ended questions were included in the site level survey asking the sites about 
the degree of flexibility and level of interrelationships of the ROC/P with other 
organizations it relates with. 

Finally, each site was invited to provide any additional comments they might 
desire concerning the subjects under study. Since no survey can anticipate every 
possible situation or response it was hoped that sites with information not easily 
fitting into one of the other categories would use this portion of the survey 
instrument to include their comments. 
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Questions on the FoIIo\v-Up 

Three questions were included in the site level survey in ouler to clarify each 
site's particular method of performing their follow-up study. The first question 
a^ed, How many courses were included in the follow-up for 1987-1988?" and "If 
this is not the total of all of the courses taught during 1987-1988, please describe 
how these courses were selected for the foUosv-up". The purpose of these questions 
was to determine if the site included in the survey all of the courses that were 
conducted during the study year or only a selected sample of courses. If only 
certain courses were included there might be a bias, intended or not, as to which 
of the courses were chosen for inclusion depending on the method used. 

Responses to these questions were virtually uniform in affirming that every 
course that was conducted during the year was also included in the foUow-up 
process. One site responded additionally that their courses which were a total of 
twenty hours or less were excluded (for e.xample, school bus driver certification 
renewal). Thin matches a general statewide practice of not considering a student 
as a candidate for follow-up unless that student has at least twenty hours of 
enrollment in a course. Only one site responded selecting a sample of courses, 
stating that courses approved by the state in an even number year were all 
included in the follow-up in one year, v\ 'lile all courses approved in odd numbered 
years were l>eing included in the next year. This appears to be a random method 
of controlling for a large number of courses being conducted and, s nce each course 
will eventually be included in a follow-up, does not present a systematic bias for 
or against particular courses over the long run. 
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The second question in this area expanils on the first one by asking "Were 
al! students in these classes included iti the follo'.v-up?" anci "If not ail students 
from the selected classes were included, how were tlie students that were included 
selected for the follow-up?" While the first question deterniined how the courses 
were selected, this question determines how the students from the courses were 
selected. Responses to this question indicated that the majority (82%) only 
included completer students in the follow-up, with the remainder including both 
completers and leavers. One of these sites indicated that they also included 
continuing students as part of their follow-up process. Ninety-three percent of the 
respondents further indicated that all of these students (whether completers only 
or both completers and leavers) were included, with only three sites indicating that 
a sampling of students was chosen. One of these three sites indicated that a one- 
third random sample was used, while another wrote about a method for drawing 
a stratified random sample (based on the criteria of home high school, gender, and 
ethnicity). 

The final question in this section was divided into three parts (named 
'Steps"). Each part asked respondents to describe their follow-up process by giving 
the following information: 

(1) How it is done (the method used to obtain the responses), 

(2) When it is done (which month(s) of the year), 

(3) Who does it (the primary person(s) responsible for the follow-up), 

(4) Number of students included in that step of the process, and 

(5) The typical response rate (given as a percentage of the students 
included) obtained at that step. 
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Prior conversations with a number of the sites uncovered that student follow- 
up sometimes occurred as a single event, while in other cases it occurred as a 
series of multiple contacts. This question was to determine how many contacts (up 
to three) were involved in the follow-up process and what each step of the process 
consisted of. 

Overall, one of the sites did not resi)ond to this question at all, and the rest 
indicated that they had at least a one step follow-up process. Twenty-nine sites 
further mdicated that they had a second step, and only fifteen sites indicated 
having a third step. Respondents to the first step seemed equally divided among 
three methods for performing the study (mail, telephone, and in class). Seven sites 
reported employing multiple methods, and only three sites stated that their data 
were obtained from visits. Most conducted this first phase of the study from 
December through February, though a few reported starting as early as October, 
and one reported ending as late as May. Sites split in terms of who performed the 
follow-up, with approximately one-half reporting that the teacher did the process 
and the other half reporting that it was an ROC/P staff member doing it. 
Responses to the portion of the question asking for number cf students included 
in that step were sporadic and unusable, with most siter not completing that 
portion of the question at all. Response rate varied widely, with eleven sites not 
responding to this item at all, thirteen sites indicating a response rate from zero 
to thirty-five percent of the students, and twenty sites indicating a response rate 
of over thirty-five percent. 

Respondents to the second portion of this question indicated using mail and 
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telephone mcthoiU as their primary niollunl^ of 5iurve>"ing. A» with the first i'tap 
the surveys wore generally conchicted from Dcceniher tlirough February. Nineteen 
of the sites responding to this phase (BS't) stated that it was ROC/P staff 
pereonnel performing this step of the foHow-up, and that the contacts resulted in 
54% if the sites with a thirty-five percent or less response rate. 

This pattern continued into the third step of the follow-up, though only 
about one-third of the sites employed three stei)s in their follow-up process. As 
with the second step, sites indicated using mail outs and the telephone as their 
primary method of performing the follow-up, with staff members being the 
personnel actually doing the task. Response rates at this step were almost 
unanimously less than thirty-five percent; in fact, most were around twenty 
percent. 

It appears from these questions that a fairly consistent pattern of follow-up 
studi^?s has emerged throughout the state. Most sites perform their follow-up on 
all of the classes conducted during the year using all of the students who completed 
the course in the sample. The best response rates are obtained during the first 
step of the process, with successively poorer and poorer rates into the second and 
third step of the foiiow-up. Data are obtained from teachers in the cla^room. and 
f^om staff personnel through mail and telephone contacts. Correlational analyses 
show that the best response rates result from data collected using in-class methods 
(r = .4241) with the vorst response mtes using mail methods (r ^ -.3960). 
Telephone methods were also related to higher response rates, though not 
significantly so. November appears to be the most successful month for obtaining 
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the highest response rates (r = .4739), though other months (April, May, July, 
August, and October) also show a high, but not statistically significant, relationship 
to increased response rates. No significant relationships appeared in terms of who 
performed the follow-up study. Even so, teachers showed the highest non- 
significant relationship to higher response rates, with ROC/P staff showing the 
next highest relationship. 

Personal contacts with the sites shed some light onto these findings. 
Virtually every site that reported teacher involvement in the follow-up process 
stated that the teachers are requested to maintain contact with all their students 
throughout the entire year. Some sites formalized this into teacher completed 
student placement cards, though most preferred to simply collect this data once per 
year from the teacher's own records. Most of this contact occurs in the teacher's 
classroom, with the teacher gaining information on the students through either the 
students themselves or through peer reports. Some telephoning was made but. in 
most cases, it was left to the teacher's discretion as to how the data was obtained. 

In conclusion, this data would seem to indicate an overall working pittern 
for ROC/P follow-up data collection. The first step for most sites would involve 
obtaming information from the classroom teacher on the student's activities. A 
second step, performed around November by the ROC/P's staff using a crmbination 
of mail and telephone techniques, would allow for follow-up on students 
unaccounted for. Additional steps could be added to the process, though the 
response rates beyond these two steps would tend to be low. 
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Questions on ROC/P Revenues 

Several questions were included in the site-level survey in order to cUu7v the 
availability and flexibility of use of revenues. Almost all of the income that an 
ROC/P receives is from state-level reimbursements . ;r student Average Daily 
Attendance (ADA",. Financial reports for prior years of operation indicate that this 
general^ amounts to anyvirhere from eighty-five to ninety percent of a site's 
income. The remainder of the income is derived from other sources. Revenue 
limit reimbursements can be used by the ROC/P for any expenditure generated, 
according to the rules adopted by the site's local governing board. Sources of 
additional income, though, may be restricted in both their availability and use. 
Should this be the case such funds may represent single opportunistic events for 
the ROC/P, unrepresentative of their usual operations. These questions were to 
determine the extent of the availability and restrictions on additional funding so 
it could be determined if special actions, such as the exclusion from the total 
revenue package, needed to be taken. 

The first questions in this section asks "Were any of the above income funds 
restricted in their use?" and "If any were, please describe the source of the income, 
the amount restricted, and what it was restricted to". Approximately 60% of the 
respondents indicated that some of their reported revenues were restricted. 
Examination of these responses showed that lottery funds were the most common 
response, with lottery monies being restricted (for the most part) to capital 
acquisitions and maintenance programs. Other restricted funds included special 
projects ^iJch as: vocational, funds for studt-nt organizations, and special grants for 
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equipment acquisitions. There did not ajipear to be any pattern to eitiicr the kinds 
or amounts of these funds and, almost lUMfornily. the amounts in question were 
relatively small (around one percent of the total revenues). 

The second question in this area was "Were any of the above income funds 
from a single-use, temporary, or non-periodic source?" and "If any were, please 
describe the course of the income, the amount involved, how many years (beyond 
1987-1988) the monies would be available to your ROC/P, if the funds are 
renewable after that period, and if the use of the fund was restricted in any way". 
Only about thirty percent of the sites indicated that some of their income was from 
these kinds of sources. Most of these were from joint powers (JP) sites, and 
indicated that the income was derived from special training programs (such as 
J.T.PJV.). Unfortunately, no site completed all of the items of the questions, 
leaving some degree of uncertainty as to the full extent of these funds. 
Nonetheless, it turned out that for those sites that did report actual financial 
amounts these amounts were very small (generally less than one-half of one 
percent of the total site revenue). One exception to this was by a joint powers site 
which reported approximately one million dollars in income beyond that generated 
through revenue limit reimbursements. This site further reported, though, that 
this money was generated as a result of yearly renewed state and federal projects, 
the renewal of which was uncertain from year to year. 

An area of current concern to the ROC/Ps visited was the distribution of 
lottery monies. Recent legislation in California allowed for the creation of a 
statewide lottery system, with a percentage of the proceeds earmarked for public 
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primary aiid secomlary tHiucation. Through not specificanj slated in the legislation 
ROC/Ps, as part of the qualified education system, were eligible to receive this 
money. Since all ROC/P monies are, however, paid to the ROC/P through the 
\ocsX districts and county agencies (as appropriate depending on the ROC/P's 
organizational type) the actual distribution of these i. onies was dependent on the 
cooperation of these diistrict and county agencies. It is possible, under the 
legislation, for the local district to retain part or all of this additional revenue in 
the district's budget, or to distribute it to the ROC/Ps with certain restrictions on 
its use. 

The final question in this area was, "Did your ROC/P receive lottery funds 
during 1987-1988?" and If you did, describe how these funds were used (and if 
their use was restricted in any way). Three-quarters of the responding sites stated 
that they did receive some lottery monies, with amounts varying from as little as 
$11,880 to $:: 13,632. Interestingly, all but one of the districts that did not receive 
lottery money were from the Coastal and Southern regions of the state. A few of 
the sites reported only receiving a portion of the total lottery funds earned by the 
ROC/P. The most common use of this funding was for instructional supplies and 
equipment, though two sites reported that lottery funds were being used to 
supplement cost-of-living salary -ncreases for faculty and administration (even 
though this was not necessarily the legislative intent of these funds). One-quarter 
of the respondents reported not receiving any lottery funds whatsoever. In these 
cases it was not that the lottery funds were not earned (since lottery monies were 
distributed during 1987-1988 on the basis of ADA generated). Rather, it was that 
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the monies were retaiiietl by the participating districts and were not distributed for 
the ROC/P's use. 

In conclusion, it does appear that restrictions on the use of non-revenue limit 
funds do exist for a significant number of the ROC/P sites. Fortunately, the 
financial amounts involved app-ar to be small relative to the overall budgets of the 
affected sites, with the possible exception of lottery funds. In these cases it 
appears that little compensation can be made in terms of an equitable 
redistribution. Conversations with several of the impacted sites indicated that 
organizational arrangements between the ROC/P and the local districts made it 
appear that other benefits had been reached by the ROC/P in exchange for the 
undistributed lottery monies. Typically, this took the form of facilities access and 
use at a reduced or nonexistent charge to the ROC/P. The conversations indicated 
that, should the ROC/P begin m; /ements to acquire these lottery funds it would 
be likely that the local districts would impose fees for the use of the district 
facilities eqjal to or in excess of the lottery amounts undistributed to the ROC/Ps. 
In this political climate most afTected ROC/Ps felt it was to their advantage not 
to pursue the acquisition of these funds but rather to continue as if they did not 
exist. Unfortunately, no data were gathered indicating whether those sites that did 
receive lottery funds paid a higher rate for their facilities than those that did not. 

Questions on ROC/P Expenditures 

The freedom to decide what objects to expend funds on is another important 
aspect in determining whether or not a given cost-effects mode! is "trulv 
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representative of the entities being nteasurLn!. The cost-effects model Uf^sumcs that 
co?itrol over er;penditiires rest^i with the ajjency })ei!ig mea.^ured. The same 
premise can also apply if a controUing influence is <hstributed equally among the 
entities or programs being measured (such as a fl£3* rate tax). It is when different 
levels of control are exercised that the model starts to exhibit trouble. In such a 
situation a cost-effects differentiation may be discovered even though each program 
may be performing equally well. The difference found may not be the result of 
differences betvireen the performance of the progran;s compared but rather between 
the extent of control each program has over its own expenditures- 
Ten questions were asked of each of the ROC/P sites in order to determine 
to what extent, if any, such external controls exist. These questions focused on 
both unique (or one time) expenditures, the same as was done in the revenues 
section, as well as on the areas of typical significant expenditures (salaries and 
benefits, facilities, equipment, and the maintenance of a reserve fund). The first 
question in this area asked "Were any of the e xpenditures given mandatory for your 
ROC/P?" and Tf any were, please describe the object of the expenditure, the 
mandatory amount, who mandated the expenditure, and why it was mandatory". 
Sixty-seven percent of the responding ROC/Ps stated that they did not have 
mandatory expenditures. Of those that acknowledged such expenditures, the most 
<X)mmon object of expenditure cited was that of indirect charges assessed by the 
ROC/P's participating school district. These charges, typically based on a certain 
percentage of either the total budget or the ADA revenues, is made by districts in 
exchange for certain district provided support services (such as: central office space 
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facilities and equipment, shared student trans^portation. and the use of district-level 
support equipment and pers(. }V Wliile the rates; of this recharge varied from 
site to site, th^ average seemed to be around eight percent of the ADA revenue 
generated A few sites also reported other expenditures as mandatory, although 
these were typically of an obligatory contractual nature (such as payments for 
facilities or equipment). 

The second question asked "Were any of the expenditure s for objects of a 
single-use, temporary, or non-periodic source?" and "If any were, please describe the 
object of the expenditure, the amount involved, how many years (beyond 1987- 
1988) the object would reoccur for your ROC/P, and if the object is expected to 
continue after that period". Like the question in the revenues section, this item 
was used to determine expenditures made by the ROC/P that should be viewed as 
both unusual and non-recurring. In this way it may be considered to exclude such 
items from total program cost considerations, as they would not represent typical 
program charges. Thirty-four (77%) of the respondents stated that there were no 
expenditures of this type made during the 1987-1988 year. Of the ten sites that 
said they did make this kind of expenditures only two reported expenditures of a 
type that did not match similar unusual and non-reoccurring funding. One of these 
two sites reporting spending an amount as repayment on a short-term construction 
loan, while the other reported an even smaller amount for the buy-out on the 
remainder of a contracted employee's contract- 
It would appear that ROC/Ps are not affected by large amounts of 
unanticipated expenditures, and that the most common mandatory expenditure is 
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in the form of liistrict-leviLnl indirect .^tjppori cluir^vs, Cojivori^aiiuns wiih several 
ROC/Ps reporting such indiroct c}u}r<:os svtM*o una}>le to deterniine both the exact 
goods and/or services that were beinjj provided for by these indirect charges, as 
well as whether or not the ROC/P could obtain similar goods and/or services 
elsewhere at a lesser cost. The impression wa? gained that the ROC/Ps regard 
these charges as part of the price of doing business, expecting a varied and perhaps 
on'y marginal return for their investment. This suggests that the costs involved 
are more determined through organizational affiliations and political negotiation 
than through cost-effective management. 

One area of concern expres^itd during the initial site visits was over 
employees' salaries. It was felt by several of the ROC/Ps that were visited that 
salaries were not f» -ly negotiated by the ROC/Ps and that, in some circumstances, 
ROC/Ps were influenced to hire more costly personnel than they might otherwise 
engage. A question was added to the survey to investigate this possibility, asking 
"Were RQC/P employees' sala ries negotiated by the ROC/P directly?" and "If not, 
please describe how employee salaries are determined, and whether the ROC/P was 
required to hire under those sal >ry limitations only 'jr if other structures could be 
used". Almost eighty percent of the sites responded that employee salaries were 
not negotiated by the ROC/P directl>. Clarification of this finding resulted in a 
clear division of the respondents. A few of the sites reported that they were either 
able to create and negotiate their own salary structures, or were an active part of 
a district(s) or the county's negotiating process. These sites give the impression 
that they are well connected with the salary determination process and are able 



Qual & EIT of CA ROC/Ps 



Page 99 CERC & UCR, 6/9/89 

120 



to exercise a degree of control in it. The majority of the respondents, however, 
stated that the ROC/P is bound by negotiations made by the participating 
district(s) or county. In mast of these cases the ROC/Ps are requiri^d to hire the 
employees according to the terms of these contracts. While this is reasonable in 
a single district (SD) ROC/P, where all of the employees are sH working for the 
same district, it can create difficulties in a joint powers (JP) or county operated 
(CO) ROC/P. In those cases the ROC/Ps report that different teachers working 
for the same ROC/P are being paid under different salary schedules, according to 
that negotiated by their primary district or county office. 

This was clarified even further through conversations with several of the 
ROC/Ps who reported that, in addition to having to use another's salary schedule 
it was not uncommon for certain districts to impose certain teachers onto the 
ROC/P. Generally, the ROC/P always reserved the right to determine the quality 
of a given instructor's credentials when considering a person for a potential 
position. Some of the ROC/Ps report that, by hiring under the district/county 
contracts, they are obligated to continue certain instructors within the ROC/P 
unless the position for which that person was hired was totally discontinued. 
Three sites further stated that their participating districts had previously required 
the ROC/P to engage a particular district teacher in order for the district to allow 
the ROC/P course to be held on district facilities. These teachers, usually tenured 
faculty on the "high end" of the pay scale, were generally qualified for the 
instructional task but were also two to three times more expensive than an equally 
qualified newer teacher would be. In these situations the ROC/P is clearly trading 
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the ability to hold the course at th.it ilislrici (nnd pcriiaps tho ccxst ot the facihties) 
for the higher cost instructor. 

The next question dealt with instructional sites, asking "Does your ROC/P 
have flexibihty in choosing inistructional sites " and If not, please describe the 
reasons that your use the sites that are used. Also, indicate if an equally suitable 
and less expensive site would be available otherwise, what it might cost, and how 
it would be used as an alternative". Ninety percent of the respondents indicated 
that the ROC/Ps do have flexibility in choosing instructional sites. Two interesting 
kinds of comments were made. In most cases the ROC/P indicated that particular 
sites are chosen for their low cost. One si^e in particular indicated that they used 
facihties made available by their participating districts for a fee of forty-one cents 
per square foot. They further state that this cost includes all utilities and 
maintenance, and that to obtain a similar space and services in a commercial area 
would more than double the cost. It appears from these and other responses that 
commercial space in the community is only rented when no other school-based 
facihties are available. This finding was mediated somewhat by the second 
interesting comment which came exclusively from county operated (CO) sites. In 
these four cases the ROC/Ps stated that courses are conducted at a variety of 
locations throughout the county-wide service area. This creates problems of 
centralization, with the ROC/Ps indicating that it is more cost-effective to 
decentralize the classrooms than to transport the students to a single, central 
facility. Regardless, however, responding sites indicated it is their preference to use 
existing school buildings and classroom space whenever possible, due to the low or 
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nonexistent costs involved. 

The final area of concern reganling ROC/ P major expenditures; is in regards 
to instructional equipment. Since no special provisions are made in the legislation 
to provide funding for ROC/P equipment acquisitions or maintenance it is up to 
the individual ROC/Ps to make provisions in their yearly budgets for such items. 
This is especially important in certain technological skills areas where employability 
depends to a large degree on the student being trained on the latest up-to-date 
equipment. 

The first of two questions in this area asked "Are the instructional 
equipment maintenance and replacement costs shared by the ROC/P with other 
agencies?" and "If the costs are shared, please describe what pieces of equipment 
are shared, who they are shared with, and how much each entity pays for the 
upkeep of the equipment". Initial site visits had indicated m a few of the ROC/Ps 
that equipment acquisition and maintenance was, at least for come items, a process 
shared between the ROC/P and local districts. It was explained that such 
arrangements would reduce equipment duplication and keep the total expenses 
down, as long as and equitable use of the equipment could be arranged between 
the participating agencies. Eighty-two percent of the respondents stated that they 
did not engage in shared equipment acquisition and maintenance. These sites 
indicated that the ROC/P purchases instructional equipment for its use only. Of 
the eight sites responding that equipment costs were shared, rr ost stated that such 
sharing occurred on an individual item by item basis. There was also a sense in 
the responses that sharing occurred more oflen when the equipment was based on 
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a district facility where the ROC/P course was also taught. Thi» would make 
sense, since the equipment would be avniia})le hoth to re^iiar district students and 
ROC/P students depending on a coordinated class schedule. Only one site reported 
sharing equipment with another non-district agency, this being the State Parks 
Authority which shared in the cost of equipment for use in a Forestry Conservation 
course (the equipment was used by the ROC/P during the conduct of the course, 
and by the state p?irks the remainder of the year). 

The second question regarding instructional equipment asked "Do you make 
long-term provisions for the upgrading and/or replacement of ROC/P instruction 
equipment ?" and "If you do, please describe the planning method used as well as 
how much money is periodically 'saved' fur this purpose. If no long-terra planning 
is done, what provisions (if any) exist for the replacement of old and out-of-date 
equipment?". Slightly under one-half of the respondents said they make long-term 
provisions for equipment upgrading and/or replacement. Responses varied from 
formalized plans (one stating a process for a multi-year plan for review based on 
the type and use of the equipment) to less formal ones (putting aside three percent 
of the annual budget to a general capital equipment replacement fund). Sites 
indicating that they did not have a long-term plan responded almost universally 
that the limiting factor to such plans was the unavailability of excess funds for such 
puk-poses. Without a stable base from which to develop multi-year plans these sites 
indicated that capital equipment acquisitions were handled on a year-to-year basis, 
utilizing such sources as: lottery funds, budget excesses, and special one-time 
funding sources. 

Qual & Eff of CA ROC/Ps Page 1 03 CERC @ UCR, 6/9/89 

124 



The final question on expenditures asked each KOC/P was to report their 
reser\'e balance as of the start and end of 1987-19SS. The California education 
code allows ROC/Ps to maintain a reserve fund for expenditures in three different 
areas: general reserve, unappropriated reserve, and capital expenditures reserve (Cai 
Ed Code 1987, section 52321). The increase or decrease in the total reserve serves 
as a fair general measure of the overall financial health of the ROC/P. Those sites 
unable to meet their expenditures with the current level of income would be forced 
to draw from their reserve, while those with excess revenues over expenditures 
would be able to increase it. Only four sites did not respond to this question. Of 
the remainder the majouty reported neither an increase nor a decrease in their 
reserve accounts. Sites reporting a change generally reported a decrease in the 
reserve averaging $24,125. A few of the sites stated that their reserve balances 
increased during the year, through these sites were clearly in the minority. 

Overall, these questions have pointed out several important facets of ROC/P 
expenditure patterns. It appears that participating districts exercise a significant 
degree of control over the functioning of the ROC/Ps, from negotiating indirect cost 
levels to their degree of involvement with teacher staffing and contracts. While far 
from uniform, it appears that almost every site is impacted to some degree by its 
relationship with participating district(s). In addition, ROC/Ps generally do not 
have single-use or temporary expenditures, though when they do exist they are 
typically offset by a corresponding revenue object. Insufficient margins in the 
overall budgets make it difficult for most sites to realistically adhere to multi-year 
replacement and acquisition plans. This results in most ROC/Ps being able to 

Qual & EfT of CA ROC/Ps Page 104 CERC @ UCR, 6/9/89 

125 



exercise actual purchases only on a year to year l)a8is, whatever their long term 
plans might be. Finally, ni()j>t sites are expeiKhng virtually every dollar that is 
generated. The across-sitcs average is to meet expenditures by reducing the 
reserve, though this reduction is typically less than one-half of one percent of the 
overall budget. 

General Comments from the Sites 

The last open-ended question on the site-level survey was a request for any 
additional pieces of information and comments that the site might feel was relevant 
to the study. Twenty-eight (64%) of the sites did not write any further comments 
m this section. Comments provided by the remaining sites centered on three major 
issues: clarifying and elaborating on responses made previously, views on state-level 
policy decisions regarding ROC/P funding, and suggestions for refinements to this 
study and suggestions for future studies. The comments that were of a clarifying 
or elaborating nature on previous question items were taken into account in the 
evaluation of responses to those items previously discussed. 

Several sites provided interesting feedback on the availability and stability 
of state-source funding to ROC/Ps. These sites reported that the complete lack of 
(in some years) and small amounts of (in others) a cost-of-living allowance (COLA) 
for ROC/Ps has made it difficult to maintain ail stafTing and support levels as they 
have been. Sites are being pushed to maintain normal increases in staff, facilities, 
and equipment expenses by reducing or eliminating services in other areas. 
Coupled with a limit on the amount of growth an ROC/P can realize during a 
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given year, and the uncertainty in actual allocations (as determined by the state 
Budget Bill), these ROC/Ps report feeling more responsible to the blate than to 
their local communities for funding. At the same time, however, they must work 
with their local school districts to insure both district support and student 
enrollment, both vital to their levels of expenditures and revenues. A strong 
message of fiscal uncertainty is conveyed by tiiese comments, with the message 
that some ROC/Ps are feeling caught between state-level budget restraints and 
local requested for changed and expanded services. 

A few sites had constructive comments on the survey instrument. The bulk 
of these reported that the ROC/P does not collect data in the way it was requested 
by the surveys (namely, course by cours**). These sites also reported that, while 
generally a lot of information is accumulated about students in and finishing their 
programs, only that data as required by the state (such as on the VE-80-C) form 
is actually tabulated and reported. One site expressed concern that a study such 
as this might show results less than favorable to ROC/Ps by relating course 
finances to course outcomes. This site indicated that many more outcomes occur 
as a result of ROC/P operation than just those being measured, and that an 
understanding of all these additional effects is necessary for a complete 
understanding. 

Conclusions from the Open-Ended Questions 

The fourteen open-ended questions were designed to obtain information from 
the sites in areas which would affect the utility of potential cost-effects findings. 
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Sites were asked about a number of areas all ^^cnerally dealing with the ii^siuu of 
flexibility in the exorcise of control over particular a^^pect.s of progrriini planning and 
operation. It is assumed that the lower the degi'ee of uniform restrictiveness the 
less generalizable cost-effects findings would be. This is posited since conditions 
between the sites would, therefore, be sufficiently different as to make it difficult 
if not impossible to either replicate or even equate. It was planned that in such 
cases where restrictions existed, sufficient information would be provided by the 
sites to enable both analysis and equative action to be taken in the cost-effects 
model. 

This attempt proved only mostly successful in that, while significant sources 
of unequal restrictions were identified, insufficient information was provided by the 
sites to determine an adequate methodology for correction. This became manifest 
in two ways. In considering the methods used for performing the follow-up studies, 
it was apparent that all of the sites employed similar techniques at about similar 
points in time. Sufficient variations between sites, especially in the response rates 
obtained at each step, suggest that there are differences that could affect both 
quality and quantity of the response data. With the subtle variations inherent in 
each site's responses, and only forty-eight sites to deal with, it was not possible 
to determine a way in which like sites may be grouped. In fact, the mo.* t useful 
organization was with all sites together, since all bites had performed a follow-up 
involving at least certain common components. 

Questions in the financial area showed that one time revenues and/cr 
expenditures were rare and usually small and typically offset each other (with a 
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one time expenditure being paid for by a one time expenditure). Since no pattern 
could be found in terms of an other than fairly rantiom distribution of these items 
they can safely be treated as normal components of ROC/P operation for 
comparison purposes. The impact of local school districts, on the other hand, 
appears to be sufficiently invasive and strong that it cannot be set aside. 
Unfortunately, while the ROC/Ps acknowledge such interactions as affecting their 
operations little was included in response to the survey questions to quantify these 
relationships. While it is apparent that local districts do exercise a high degree of 
control over ROC/P autonomy it is not possible, do to these incomplete responses, 
to relate different organizational relationships to potentially different cost and/or 
effects. As important as this appears, it will necessarily have to wait for a future 
investigation in that area. 

Conclusions from the Site Visits 

Sites were chosen on the basis of three criteria: (1) geographic distribution 
throughout the state, (2) organizational type (Single District, Joint Powers, and 
County Operated), and (3) kind (ROC, ROP, and ROC/P). The visits were 
conducted from October, 1988, through March, 1989. Interviews concentrated on 
three m^jor areas of ROC/P operations: EnrolIment/Attendance/Follow-up, 
Budgeting and Fiscal Management, and Program Planning and Operations. 

Data obtained during the different site visits tended to reinforce that 
communicated through the site-level surveys. All sites visited have internally 
consistent methods for student attendance accounting. Few sites produce 
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attendance reports beyond those re(|uircd for ADA determination purposGs. There 
were diiTerences between sites in whether attendance was recorded by student 
presence or absence, as well as how frequently the attendance records were 
transmitted from the teacher to the ROC/P central office. This contributed to 
diflerential methods for addressing student absence issues, with some sites 
preferring to leave all attendance issues to the local high school (except for 
nonconcurrent students) and others wishing to handle absence follow-up for all 
ROC/P students regardless of local district policy. 

Considerable variation was found in ROC/P follow-up methods. All sites 
attempt to comply with the State guidelines (VE-80 forms B and C) for following 
up on students, though there are significant difTerences in terms of both student 
category definitions and data collection strategies. "Continuing" students are 
generally defined as those remaining in the ROC/P course into the next reporting 
period, though some sites extend that definition to include any concurrent student 
not completing high school (since they might return to ROC/P in the next year). 
The requirements for a "completer" vary, with some sites employing strictly 
academic criteria and others including students who obtain employment in the 
field trained (regardless of the level of academic achievement). Additionally, since 
the State does not require 'leavers" to be included in ROC/P follow-up studies, few 
sites include them as part of the study group. 

Follow-up sampling procedures differ as well. Most sites sample all students 
classified as completers in all courses. A number of sites sample all or some of 
the students in a selected sample of the courses. Determination of a student's 
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employmont status is generally based on whether they are employed at the time 
of the foUow-up study, which may occur anywhere from zero to six months 
following course completion. Some sites check to see if the students were ever 
employed since completion (regardless of their curren work status). Students who 
could not be contacted are generally grouped into the category "status unknown", 
though some sites divide the status unknown students into the other follow-up 
categories based on known respondent's percentages. 

Especially troublesome in any effort to use student follow-up data is the fact 
that the seven follow-up categories based on the VE-80 form are neither complete 
nor mutually exclusive. This leads to different methods of categorization of some 
students (such as when a student is both working part-time and continuing his/her 
education). There were many different methods used to perform the follow-up 
survey (including instructor contact, mail-outs, and telephoning) and a wide time- 
frame over which the follow-ups are performed. All of these issues contribute 
significantly to the difficulty of providing accurate between-site data analysis. Only 
a few sites employ follow-up results for their own meaningful program analysis. 
Most collect the data merely to fulfill the VE-80 requirements. 

Most of the sites visited do not have structured plans or methods for periodic 
course reviews. While all recognize the mandate for such reviews, in most cases 
courses are reviewed only when significant problems become evident. Of the sites 
engaging in structured periodic reviews most apply criteria that extend beyond 
course mcome, costs, and job placements, including: teacher and student evaluations, 
student final grades, and advisory committee inputs (to name a few). 
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Conclusions 

This study has produced significant findings in a number of important areas. 
First, the development of California's ROC/Ps has occurred within a process that 
is dynamically responding to the changing needs of both the stude.it clients served, 
the participating local school districts, and the State policy makers. ROC/Ps have 
evolved by serving the needs of several different important interest groups. As a 
result, program planning and evaluation are influenced by organizational and 
political factors as well as by fiscal and legal constraints. 

Second, simple cost comparisons, outcome analyses, and even cost-effects 
studies cannot fully describe the complex factors influencing ROC/P program 
performance. Sites where interdistrict harmony and home-school integration are 
dominant concerns necessarily generate a different mix of student services from 
ones where the primary focus is on training adults to fill labor market shortages. 
Moreover, differences between state-level goals and local district-level goals subject 
program managers to complex cross pressures. 

Perhaps the most important lesson from the research project is the extent 
to which it documents problems of uncertainty and instability confronting ROC/P 
program managers. Across a broad range of fiscal, organizational, and program 
dimensions, CAROC/P managers face rapidly changing conditions and an uncertain 
future. 

In the ar ea of Fiscal Support , the data reveal year to year 
fluctuations that have reached as high as 30% of the base 
revenues. 
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In the area of Romilnfinns, CAKOC/Ps have faced rapiil chan^^es 
calling for the coIlGction and analysis of data which is 
expensive to gather and hard to synthesize into 
meaningful program guidance. 

In the area of Accountnhillfv, pressures for documentation for 
cost-effective programs have risen sharply while assistance 
and resources have improved only slightly. 

In the area of HiPnt Recruitment . CAROC/P programs are 
confronted by the need to attract students to a sure 
program income which is controlled by enr'>Ilmeiit in 
courses rather than by placement in jobs. 

In dealin g with Cooperating Districts . CAROC/Ps fmd that much 
of their autonomy and independence is absorbed by 
preexisting teacher's contracts, limited facilities, and 
political constraints on their program options. 

In responding to Chan ^ng Market Conditions . CAROC/Ps find 
themselves responsible for trucking rapidly changing job 
opportunities but have no specific resources or training 
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appropriate to this res]>«in?;il>ilily. 



In btiilding a Sup port Cnn^titucncv . CAKOC/P manag^ers find 
themselves relatively isolated and lacking in strong 
advocates among state policy makers, local district leaders, 
or any other major education interest group. 

These problems of uncertamty and instability call for substantive actions 
aimed at creating a stable environment and broad-based stipport for CAROC/P's 
role as a leadership agency for vocational education in California 

If California's Regional Occupational Centers and Programs are to be fully 
understood, any system of review and evaluation will have to take these factors 
into account. Clarifying the methods of evaluation to be used for ROC/Ps, as well 
as stabilizing and increasing the funding, is only part of a solution. The remainder 
c ernes from understanding the economic antl social environment in which ROC/Ps 
yxist and must function. Results from this study clearly demonstrate the 
importance of developing 2 broad consensus on ROC/P goals, and the building of 
a Management Information System utilizing consistent data definitions and 
standardized data reduction and analysis techniques. 
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Recommendutions lot* Further Action 

The CAROC/P study just coniploted hi«;hlightsS a critical need for butter 
linkages between the several missions served by regional occupational centers and 
programs, and the ways in which supportive management information is gathered 
and analyzed. The Management Information System component of this study 
ideniified issues affecting ROC/P stability in seven distinct areas: fiscal, regulatory, 
accountability, client, cooperating districts, market conditions, and constituency- 
Information collection, analysis, and utilization must be aimed at stabilizing 
and strengthening ROC/P performance in these areas by reducing the uncertainty 
and vulnerability currently plaguing program managers. The lack of information, 
as * as the difPculty in applying information regarding the marginal utility of 
various policy and program options, only confounds the issues. 

A Continuation of M.I.S. Support 

The first and most important step is to assist CAROC/P to understand the 
results of the current study, and to develop management information systems that 
enable them to document the existence of specific elements used to secure program 
stability and to deliver dependable training services. This M.IS. development 
involves both a staff development process (enabling CAROC/P members to build 
local capacity) as well as further research and development work (to help identify 
specific data elements and analysis procedures). 

To be successful in future planning and operations California's ROC/Ps must 
address themselves to understanding the parameters of the seven areas mentioned 
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above. Knowing about them is only the beginning; planning for effective change 
Inquires that managers appreciate the cumponent elements dris-injj each area. 
Future M.I.S. support work would begin with disseminating the findings of this 
research to the statewide ROC/P units. Feedback would be elicited from managers 
as to the component structures of each area, including concrete ways in which 
these are both measured and used in current planning and operations practice. 
Results would be tabulated and compared in order to identify potential type, 
regional, or statewide patterns. A final report would summarize these seven areas, 
their component structures, and ways in which the ievel of instability in each is 
being successfully reduced through particular intervention practices. It is our 
strong recommendation that such an effort, whether performed by us or by others, 
be considered as a cornerstone to further work. 

A Study of Exemplary Practices 

One of the best ways to improve organizational performance is to model 
operational programs ailer those already proficient at the task. This applies to 
ROC/Ps as much as any other educational agency. The twenty-two site visits of 
the study just completed havr* shown the full range of effective ROC/Ps. It is 
dear that some ROC/P sites are much better than others at performing different 
management tasks, operating different courses, and generally providing services. 
These "exemplary" ROC/Ps could be used as role models for others in the state, 
with their methods studied and made available for replication. 

A study such as this one would require the identification and cooperation of 
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the best ROC/P sites in California, as detenninLHl along several dimensions of 
interest. These sites would be visited for a period of up to one week each for an 
in-depth investigation of their outstanding operations. Findings from the sites 
would be aggregated according to the dimensions outlined. The final report would 
consist of a h^dbook detailing the factors contributing to dimension success, with 
suggestions on how other sites might achieve them. 

The M.I.S. planning resulting from such an investigation would center 
around the items identified as contributing to high program performance. Part of 
the handbook would be ways in which sites could collect and analyze information 
to monitor progress towards specific program goals. Since role models would 
provide the standard of excellence, achievement levels and progression rates could 
be determined. 

A Study of Cohort Groups 

A further step in stabilizing CAROC/P programs and services is to document 
the importance of current services to the individual students served. The study 
just completed defines the problems of program and policy stability, but cannot 
document the valus of services to students. To show state and district policy 
makers the value of the services provided requires information on the students 
themselves. 

The question is whether the regional vocational education provided students 
is truly effective in assisting these students into employment, further education, 
and other benefits. The key to this question, of course, is: more effective than 
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what? Wliile statistics of completer rates, job placements, and the like can be 
calculated and quoted there are signitlcant questions as to the comparability of this 
data, as well as it is still unknown to what degree the students would have 
achieved these goals had the students not participated in ROC/P education. 

To answer this question requires not only collecting data on the students 
who participate in ROC/P education, but also of students who did not. This would 
require the cooperation of not only selected ROC/Ps, but also several participating 
school districts in order to obtain the necessary cohort data. Specific outcomes 
would have to be carefully quantified, as well as the methods and time frames 
used to collect the data on these outcomes. Since immediate outcomes are often 
not those of long-term interest, the study would have to be both cross-sectional (of 
selected demographic and subject-area characteristics) and longitudinal (over the 
course of several years follow-up). 

A cohort study would provide clear and compelling answers to many 
questions. Primarily, it would demonstrate the utility of ROC/P vocational 
education agains^ both no vocational education and other alternatives available to 
high school students. The study would provide real- world comparisons of different 
methods of achieving similar outcomes. It would also allow for a complete 
quantification of the costs involved in each methodology, to truly show the cost- 
effectiveness of ROC/P vocational education. 
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Appendices 
Appendix A: Site Visit Protocol 

Introduction 

Self, CERC, and UCR 

Research study overview 

Coftt-effectiveness analysis of selected courses/sites 

Piupose for this visit 

Discover in which course cost and enroIIment/follow-up data is kept 
LlanQr definitions for costing and completion/leaver categories 
Find sources of local variation, an{l concerns of local units 

Obtain overview of site 

ROC/P type, area served, numbers and subjects of courses offered 
Typi^ student enrollment by course by sex by con/nonconcurrent 
Chief administrative contact person, phone, address, assistant 

Enrollment data 

VE-80-B form in by August 31st? Accurate? 

Definitions of: concurrent v. nonconcurrent (v. other?) 

„ _^ . continuing v. completer v. leaver (v. other?) 

Jactonng course length and student total attendance 

"Getting a job" v. "academic completion" (v. other?) 

Anyone not reported on the form? 

Could repUcate of a single course be reported? 

Follow-up data 

VE-80-C form by January 31st? Accurate? 

How sampled: all. random, stratified, other? 

n *- ^ n, sample reported, or applied as if to all? 

Ueiimtions of follow-up categories 

Follow-up on "completers" only, or both completers and leavers? 
Fenod of time when sampling is done? Criteria for "being employed^ 

Cost data 

Which "J" forms reported? When? Accurate? 

Data available on a course-by-course basis? Paper or Electronic? How? 
hnportant pbjecte of expenditure" (object code, EDP number, form) 
Diflerentiation of "direct" costs and "indirect" (overhead) costs? 
Appropriation of revenues on a course-by-course basis possible? 

Wrap up 

Any additional info that might be helpful. 

iSaJdfy^? ^ participate/assist with further data collection/analysis 



Qua! & Eff of CA ROC/Ps 



Page 130 

151 



CERC @> UCR, 8/9/89 



Appendix B: Connnnation Letter for a Site V'isit 

[Director Name & Title] [Date] 

[ROC/P Name] 

[ROC/P Street Address] 

[ROC/P City], [ROC/P State] [ROC/P Zip] 

Dear [Director Name], 

I wanted to thank you for being able to meet with me on [Meeting Date] at 
[Meeting Time] at your site in order to discuss the data collection and analysis 
aspects of the California Association of Regional Occupational Centers and 
Programs (CAROC/P) Research Study. — 

As I mentioned over the telephone I am one of the researchers from the 
California Educational Research Cooperative (CERC), a unit of the School of 
Education at the University of California at Riverside, that will be working on this 
project. This study involves an investigation of the legislative and fiscal history of 
Califomia*s ROC/Ps since their inception in 1963, as well as an evaluation of 
ROC/P cost-effectiveness. It is this second part that I am interested in meeting 
with you about. 

The State of California collects, with the "J" series fiscal forms and the VE- 
80 series vocational education forms, information on the cost and outcomes of 
ROC/Ps. One of the purposes of this study is to go one step further than the 
mandated reporting by looking at the costs and effects of individual classes within 
many different ROC/P sites throughout the state. The purpose of our upcoming 
meeting is so that I might become acquainted with your site's methods for 
collecting, maintaining, and reporting individual course cost (both direct and 
indirect) and effect (follpw-up) data. 

Naturally, any additional staff that would be able to assist with this process 
would be extremely welcome. You had mentioned [Bus Mgr Name], Business 
Manager and [Stud Rec Name], Student Record Manager, as individuals who would 
be able to make significant contributions. I would welcome everyone who might 
be able to shed some light onto this topic. 

Again, thank you for takirig the time out of your busy schedule to 
accommodate me as the research study. I look forward to our meeting. 

Sincerely, 

Jeffrey B. Hecht 
Research Fellow 
CAROC/P Research Project 
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Appendix C: Thank You Letter for a Site Visit 



[Director Name & Title] 
[ROC/P Name] 



[Date] 



[ROC/P Street Address] 
[ROC/P City], [ROC/P State] [ROC/P Zip] 

Dear [Director Name], 

I want to thank you and your associates for taking the time during the past 
week to meet with me concerning the data requirements for the California 
Association of Regional Occupational Centers and Programs Research Study. My 
visit to your site, and the information that I obtained during our [Mtg Lengtii] 
meeting, is proving to be extremely valuable in formulating the requirements for 
further data collection and analysis. Without our pre-data collection mterviews 
such as these certain assumptions about the uniformity of infoimation meaning, 
gathering, storage, and retrieval would go untested, which could lead to interesting 
(but unusable) results in a study such as this if there indeed proved U be 
significant inter-site differences. 

As I mentioned during my visit I will be back in touch with you during the 
first part of January, 1989 regard the actual data collection process. In the 
meantime if you should think of any further items that would be important to the 
study please do not hesitate to contact me. Thanks again. 



Sincerely, 



Jeffrey B. Hecht 
Research Fellow 



CAROC/P Research Project 



Qual & Eff of CA ROC/T« 



Page 132 



CERC @ VCR, 6/9/89 




153 



Appendix D: Course-Level Survey Instrument 
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CAROC/P Research Study 
Course FoUow-Up 



Ctencral Instnietlong 

nUs part of the survey covers course t*nn)llinent and 
follow-up data. Please complete one form for the ctiurso 
iksJs^iated on the labd attached to the right Complete all 
ftems on the attached form, writing "N/A* or "Unknown" on 
aiqr lines for which data is not available. Rqjort only those 
amnbeis that appfy to the 1987-1968 ofiTcring of thfe courae. 

If multiDte sections and/or locations of the same 
course wrae condvKled total them together and rqwrt the 
ti^ Do howevn-, report figures on an cztirc program 
of whkh this course mij^t be one of several Report data on 
all sections of this one course only. 

Follow-tiP Data 

Report daU used by your ROC/P to identify 
Completers*. Entiar the total number of students clasKscd as 
ewnplBteni for this course by filling in the appropriate 
Bumber nest to «»ch statcmMit. If all of the completers for 
tiiis courw fiall under only one of the statements then write 
the total number there, writing "N/A" in the other spaces. 
The ^ateo^t Total Number of Completers' should be the 
nun the three previous values, and should i^jrce with the 
total number of completers reported on yoinr 1987.1988 VE- 
fionn for thu course 

Enter the total number of "Leavers" on the next line. 
RQ>ort the number <rf both completers and leavers that were 
induded in the follow-up procc^ for this course only. Total 
these two numln^rs in the final line. 

Status Categories Data 

The Status section asks you to dctaC the follow-up 
figures for all students who enrolled in this course, both 
completers and leavers. If ^ur follow-up was on both 
completely and leavers please write the appropriate numbers 
on both sides of this section. If only completers were 
fdlowed-up, you should only write in the 'Complete rs" section 
(write in T^/A" in the "LCTvers- section) agd write the words 
*Only Completers Fdlowcd-up On" at the bottom of the page. 

The items in bold type arc the reporting categories 
reported on your 1987-1988 form for this course. 

Begin by copying the vaJucs you rq>orted under thase 
headings on the VE-SOC to the appropriate lines on this 
fbnn* 

If you have more detailed foUow-up information 
available, please enter appropriate figures onto the spaces 
proviifed The fWlowing definitions will help darify the 
owning of these addiUcnal cati^rics. Information from 
thme more detailed foUow^tp categoHea will be 
extremely valuable if you have it available. Should 
yon be using follow-up categories not listed please foel free to 
add them onto the sheet, but be mre to provide a written 
expl anation ctf what your categories mean. 



Status Categories 

Full Time Employment: Employed Full Time in the fldd 
trained . 

Part Time Emplojnmeiit: Employed Part Time in th e flcdd 
trained ^ 

Unknown Employment: Employed in the field trained, but 
it is unclear whether it is Full or Part time 

Full Time Employment: Emplcyed Full Time in a 
non related field . 

Part Time Employment: Employed Part Time in a 
nonrelated field. 

Unknown Employment: Employed in a nonrelafed fidd» 
but it is unclcmr whether it is Full or Part time. 

ILS, Without P/T Employment: Returned to h^h school 
and not cmplcyed in any fidd. 

ILS. With P/T Employment: Returned to high school and 
employed in any field. 

II.S. Emplojrment Ui^nown: Returned to high schod but 
employment status is unknown. 

AdiUl Voc Ed Training: Undertaking additional vocational, 
technical, or trade rclated education. 

College/University: Attending a college or university. 

GED Pi^ogram: Puj^uing a GED. 

Military Service: Enlfeted In a branch of the militaiy. 

Can't Contact: Whereabouts unknown (bad-address, 
disconnected phone, momi away). 

Unelasslllable: Student contacted but cannot be dassificd 
bemuse re^nscs are vague or unintcdl^ble. 

Non-Response: Contact unsucccsrfUI Iwjcause the student did 
n(rt respond (letters not returned, phone not 
answered, student uncooperative). 

Unable to Work: Wants to worit but is unable to because 
of physical, family, or other hardship considerations. 

Unwilling to Work: Docs no| want to work and is not 
.seeking employment. 

Other: Any other mxson (please darify if used). 
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Follow-Up 



Course FoIIow-Up 



Employed in any field but did not complete the course s academic requirements 
Employed in any field and did complete the course's academic requirements 
Not employed and did complete the course s academic requirements 
Total Number of Completely 

Total Number of Leavers 

Number of completers included in the follow-up 

Number of leavers included in the follow-up 

Total Number of Students Included in the Follow-up 



Status Categories 



Completers 





Con 


j Noncon 


Total 


Cpn 


Noncon 




Employed in Field Trained 
Full Time Employment 












1 Totj^ 


Part Time EmDiovment 














Unknown EmDiovment 














Total Emoloved. Field Trained 














fUnpioyed Nonrelated Field 
FuU Time Emolovment 














Part Time EmDiovment 














Unknown EmDiovment 














Tot^ EmplQY^. Nonrelated, 














Pursuing Additional Education 

as. Mthout P/T Employment 














H.S. With P/T Employment 














H.S- Employment Unknown 














Addtl Voc Ed Training 














L/Oiiege/ umversitv 














GED Program 














^ Total Pursuing Addtl Ed 














Military Service 
Total Military Service 














Status Unknown 

Can't Contact 














Unclassifiable 














Non-ResDonse 














TotaJStatus Unknown 














Unempl^ed, 2^king Employment 














Unable to Work 














Unwillinff to Work 














Other 














Tc^tal Other Reasons 














Grand TotaL 















Leavers 
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R«i4s«d Januaiy 18, 19^ 
Univmlty of CuiUmtla nt RWmldb 



CAROC/P Research Study 
Course Finances 



General Inst ntet ions 



This pan of the sv rwy covers n?vcnut* and 
expenditure data on the spccitk ROC/P course 5hown on the 
labd attached to the right Please complt\j one form for 
eodh cmuBc draignatcd (label attached to the right). 
Complete ag items on the attached form, writing ^/A" or 
*1Unknown" on ai^ lines flor which data is not available. 
Bepwt only those nombers that apply to the 1987-1988 
dS^ring of this course 

If multiple sections and/or locations of the same 
ctHixse were conducted total them tc^i^er and report the 
total Do sot, howevCT, report figures on m entire prt^m 
of whidh this roursc m^ht be one of severaL Report data on 
aU secttoM of this one course only. 

ADA InitrncHons 

Re port the head-count enmUment the Full Time 
Equi^mlent (PTE) svemre dsflv attendanm (ADA), and total 
number erf student instructional hours for this course. Enter 
cxmaurent and non^conciurrent students »?parately, the total 
the two groups on the Une provided 

Revenues Instmetieiys 

We are interested in all nr/cnucs thi»t ct>uld be 
attribute to this course. The categories to I reported 
ftOlow the J*30(»*ROP form, except that whOc the J-SCMJ-ROP 
reqwsta information on the imtire ROC/P we would like 
xeveni^ information as it applies to thas cour^ only. 

Please be carcftil to break out the "Sales Prom the 
Operation of a Business" and Tecs & Contracts* from the 
cat^gwy •Othcs' Local Revenues*. Report only those other 
revenues that att? not sales and/or fees on th-? line 'All Other 
liOcal Revenues\ 

If there are other sources of revenue not accounted 
for in the n^ar categories please include them on an 
't)tber'' lhae. Detail the rourcc of each revenue amount, its 
EDP line number (if known), and the amount earned 
Isdude a short eiq>lanation of what the revenue j^uree is 
how ft was obtained, whrthcr it is one time only or 
contfaittinR and if there were restrictions on how it could be 
used If grants, contracts^ or othw mxitccs of revenue 
provide incon^ for more than one course (eg« an equipment 
m general support grant), apportton this inromo and show 
the aaounrnt attributable to this particular course. 

Total Revenues' should be the mm of aU the lines, 
and should equal the total amount <rf rcmme directly 
i^»ierated by and attributed to this coutks. If you were to 
repeat this proce» for each and evi^ onirse taught at vour 
ROC/P the totals ct all these sheets should equal the total 
nvmm generated for the ROC/P. There should be no 
melitdeil income* 



Expenditures TnstrMCtions 

Detail the caui^speciHc expenditures for this couiBe 
Include only those expenditures ^at can hm identifiedi 
as belonging specifically to this counm. Do report 
on th» form expenditures made by the ROC/P in s«piK)rt of 
many courses which canned be directly attributable to this 
individud course {eg, Superintcndimt's salaxy, central 
building, central office supplies). A separate form has been 
provided to collect ROC/P-lcvd information, 

IMcnnine the aumbw of Full Time Equivalent (PTE) 
teachers and instrucrtional aides, as well as all other 
certificated and dassifled staff ddUhrering services to this 
course, and rqnirt the amounts on the blanks provided 
Report tethers' salaries on a separate line from all other 
certificated salaries, and sq)arate Instructkmal aides* salaries 
from all -other classified salaries. When a person's sole job 
in for this particular course rount their entire annual salajy. 
When a person is split between this course and one or more 
other job res^nsibilitics determine the total percenti^ of 
their time expended on this course and multiply that by their 
annual salary. Employee bendlts apply only to those 
perscm(s), or pcrccn^e of per9on(s) time; rapended in 
support of this course. Prason^s employed by the ROC/P 
without a direct and identifiable link to this course should 
not be reported on this form. 

Separate Total Services and Other Operating Expenses* 
by Tacilities", "Equipment", and "All Other 
Serviccs/Operatmg Expenses". When facilities and equipment 
are shared between several coursi^ compute the parentage 
of ph>'sic3il anm and/or time uwd for this course then 
muJtiply that by the annual co^ For both facilities and 
equipment check the appropriate Une (or several lines if more 
than one type is tppropiiate) snd write in an anwrnnt 
iq>ending on whether it is: 

Free: arailable without any cost. Enter zero for cc^t 
District: provided for a fee by a local school The amount is 

the nuaJ charge for this course. 
Rent: ; Jable throi^h a lease or rental agreement Enter 

the annual amount paid for the rrot 
Own: owned by the ROC/P. The amount is the annual 

mortgage or bond paid If fUlly paid for, enter zim 

Detail other items of expenditure on an "Other" lina 
Total Expenditures- should be the sum of all the Ihios, and 
should equal the total amoimt of esqpenditurcii incurred by 
this mum}. If you were to r^K»t this process for each and 
e\eiy course at ycmr ROC/P the totals of all these sheets 
should equal the tcAal direct costs for the ROC/P. When 
added to the ROC/P-lcvd indirect/support caito, this total 
should be the total exi»mditurcs for the R<X:/P. Th^re 
should be no excluded eirpendlturM. 
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Course Finances 



During 1987-1988, 



Concurrent: 

Non-Concurrent: 

Total: 



sections of this course were taught at 

Enrollment ADA Stud Inst Hrs 



different locations. 



Actual Dollars (EDP) 

Total Revenue Limit Sources . (200) 

Total Federal Revenues (280) 

Total Other State Revenues (450) 

Total Other Local Revenues 

Sales From Operation of a Business (483) 

Fees & Contracts (508) 

All Other Local Revenues 

Total Prior Year Revenue Adjustment (001) 

Othen ( • ) 

Othen ( ) 

Total Revenues (599) 



Expenditures 



Total Certificated Salaries 

Teachers' Salaries: . FTE 

All Other Certificated Siiarios: . FTE . . . 

Total Classified Salaries 

Instructional Aides* Salaries: . FTE 

All Other Classified Salaries: . FTE 

Total Employee Benefits 

Total Books & Supplies 

Total Services and Other Operating Expenses 

Facilities: Free District Rent Own 

Facilities CJost . 

Equipment: Free District Rent Own 

Equipment Coit 

All Other Services/Operating Expenses 

Total Capital Outlay 

Total Other Outgo 

Total Direct Support/Indirect Costs 

Total Prior Year Expenditure & Other Adjust . . . . 

Other: 

Other: 

Total Expenditures 



(600) 



(630) 



(729) 



(780) 
(846) 
(855) 
(002) 
( ) 
< ) 
(599) 
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CAROC/P Research Study 
ROC/P Total Survey 



Directions 

The next several pages ask for information relating to several aspects of your 
ROC/P. The areas include: basic information, student enrollment and follow-up 
methods and total outcome, income and expenditures, and general comments. 
Please drculate this survey to the appropriate people within your organization for 
completion of the different sections. Do not separate the pages, since only the 
front most page is labeled with your site identification. If you should have 
additional comments on any of the items please feel free to include them on the 
final page and/or to attach additional sheets. 

Thank you for your cooperation and assistance. 

Basic Tnformation 

County: District: School: 



ROC/P Name: 



Mailing Address: 
Street Address: 



Person to Contact: Telephone: ( ) - 

Type (drcle one): County-Operated Joint-Powers ^:in^e-District 
If County-Operated or Joint-Powers, how many school districts ser/ed? 

Number of different courses (ROC/P ID Numbers) taught in 1987-19158: 

ROC/P Location (drcle one): Urban Rural Mixed 

m 

Size of population base served: 

Economic status (circle one): Lower Middle Upper Muted 

CAROC/P RoMareh Siuciy Page 1 Revisad Januarv 20. ifiSA 

B*««Uo»id B«..n* Co«p^Uve Un^Hi^C,^^.!?^^ 
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CAROC/P Research Study 
ROC/P Total FoIlow^Up 



General li^strttctions 

Thm part of the sunw .sccki$ cnn)Umrnt 
and follow-up data for your entire Re^ntmal 
Occupational Center or Pn>Kram- Plea^* repDrt the 
total for all rourscs conducted during lf)874J>«S. 
Total all actions and locations of each cnunie 
conducted. Complete alj items on the attached 
form, writing "N/A* or "Unknown* on any lines for 
which data Sa not available. Report only thase 
naml^B that apply to the 1987-1988 reporting year. 

Follow»wp Data 

Report daU used by your ROC/P to identify 
Xompleters^ Enter the totSal number of »tudenu 
classed as completers fbr all cour^ by niHng in the 
appropriate number next to each sUtement If all 
the completers for all courses faU under only one 
of the atatonenta then write the total number there, 
writing T^/A* in the other spaces. The statement 
"Total Number of Completers* should be the sum of 
the three previous values, and should agree with the 
total number of completers reported on all of your 
1887-1988 VE^ forms for this ROC/P. 

Enter the total number oi "Leavers" on the 
nart line. Report the number of both completers 
and leavers that wen? included in the follow-up 
process. Total these two numbers in the final line. 

Statwa Data 

The Status section asks for detailed follow^ 
up figures for gJl students, enrolled In all counn?, 
completer or Icawr, If your follow-up included both 
completers and leavers please write the appropriate 
numbers on both sides of this section. If only 
completers were followed^up, enter data only in the 
"Completers* section (write in "NVA" in the ^'Leavers'* 
section) arjd write the words 'Only Gimpleters 
Fdlowed-up On* at the bottom of the page. 

The itcmis in bold type are the reporting 
categories reported on your 1987-1988 VE-SfC 
forms- Bc^n by totaling the vaJuisi you reported 
under those hradii^ on the VE-aX3 forms for all 
courses to the appropriate lines on this form. 

If you have more dcUiled follow-up 
information available, plea^ enter appropriate 
figures onto the spaces provided The following 
definitions will help clarify the meaning of these 
additional categories. Information tr^m these 
more dbtatled follow-itp categories will be 
extremely valuable id you have it available. 
Shmild you be using foUow-up c^tc^ries not listed 
please fed free to add them onto the sheet, but be 
sure to prwide a written cynlanation of what your 
categories mean. 



Stntim Categories 

Full Time Eniplopnent: Empioyed Fn I I Time in 
the field trnmH . 

Part Tl nie £nipIo5^nent: Employed Fart Tinie in 
th e field tmincd . 

Unknown Employment: Employed in the fidd 
trained, but it is unclear whether it is Full 
or Part time. 

Full Ti me Employment: Employed Full Time in 
a non-rriated field . 

Part Time Employment: Employed Part Time in 
a nnn»related field 

Unknown Employment: Employed in a non- 
related field, but it is undear whether it is 
PuU or Part time 

U.S. Without P/T Employment: Returned to high 
school and not employed in any field 

U.S. With P/T Employment: Returned to high 
school and employed in any fidd 

U.S. Employment Unknown: Returned to higli 
school but employment status is unknown. 

Addtl Voe Ed Training: Undertaking additional 
i^>cataonaU technical, or trade related 
eduoition* 

College/University: Attending a college or 
universit}'. 

GED Program: Pursuing a GED. 

Can't Contact: Whereabouts unknown (bad-addrcss, 
disconnected phone, moved away). 

Unclasslfiable: Student contacted but cannot bo 
dassified because responses are vague or 
untntdligibla 

Non*Response: Contact unsuccearful bccauM the 
student did not respond Uetters not 
returned phone not answered student 
uncooperative). 

Unable to Worh: Wants to woric but is unable to 
because of physical, family, or other 
hardship consideratbtis. 

Unwilling to Work: Does not want to work and is 
not taking wipIoymenL 

Other: Any other reason (please clarify if used). 
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FoUow'Up 



ROC/P Total FoUou-Up 



Employed in any field but did not complete the course s academic requirements 
Employed in any field and did complete the course s academic requirements 
Not employed and did coi .plcte the course s academic requirements 
Total Number of Completers 

Total Number of Leavers 



Number of completers included in the follow-up 

Number of leavers included in the follow-up 

Total Number of Students Included in the Follow-up 



Status Categoric 



Completers 



jcaxipioyvo m rieio iraineci 
Full Time Emolovment 


Con 


Noncon 


Total 


Con 


Noncon 


Tot^l 


jran nine rjnpioyincnt 














uuKnown juinr lovmen t 














fempiQV^, Field Trained 














iianpioyeci inoiireiaieci ricici 
Full Time Employment 














c^otl liiut; CiUipiuymenL 














Unknown EmDiovment 














— 1 <ytffl ^ynplov^i Nf>nr^latgd 














x^ursuing Aaaiuonai JCiCiucation 

n.a. wiuiouc r/1 £4mpi03'ment 














HS Vinth P/T Fmnlrivmont 














H.S. Employment Unknown 














Addtl Voc Ed Treinin^r 














College/University 














GED Program 














^ Total Pursuinir Ah Ul Ed 














Xklilitary bervice 
Total Military Service 














Status Unknown 

Can't Contact 














Undassifiable 














Non-ResDonse 














— TpM Statwg ynknpwn 














Unemployed, Seelcing Employment 
Total Unemnloved^ Seeking 














Other Reasons 

Unable to Vork 














Unwilling to Work 














Other 














Total Other Reasons 














Grand TotaL, 















Leavers 



CABOC/P B«saarch Study 



Revised Jatiuary 20« 1980 



CAROC/P Research Study 
ROC/P Total FoUow-Up 



(l)How many courses were included in the follow-up for 1987-198S? 



If this is not the total cf all of the courses taught during 1987-1988, please describe 
how these conrses were selected for the follow-up. 



(2) Were all students in these classes included in the follow-up? YES NO 

If not all students from the selected classes were included, how were the students 
that were included selected for the follow-up? 



(3) Please describe your follow-up process, being as specific as possible. For each step in 
the process please detail: 

a) how it is done (the method used to obtain the responses) 

b) when it is done (which month of the year) 

c) who does it (primary person responsible for the follow-up) 

d) number of students included in the step of the process 

e) typical response rate (percent) obtained from that step 

For each step include appropriate comments, as necessary, to assist in our understanding 
of what IS done at that step. If possible include with this survey sample copies of any 
survey instruments, mail-outs, and the like that are a pan of your ROC/P*s foUow-up. 

Space is provided on the next page for your responses. 



CABOC/P Besetureh Sfcuify 
thMantim lE<h»c«Uoit«l RnMmrrh Cooperative 
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R«vis«d January 20, 1888 
Vntvvnatjr CaIIfi»mfai mt Hl««nld« 



CAROC/P Research Study 
ROC/P Total FoIIow-up 



Method 



Wlien Done 



Who \}oes U 



# Studs Response 



Step I 



Mail 

Telephone 
In Class 
Visit 



Jan 

Feb 

Mar 

Apr 

May 

Jun 



Jtil 

Aug 

Sep 

Oct 

Nov 

Dec 



Teacher 

Administrator 

Staff 



% 



Comments: 



Step II 



Mail 

Telephone 
In Class 
Visit 



Jan 

Feb 

Mar 

Apr 

May 

Jun 



Jul 

Aug 

Sep 

Oct 

Nov 

Dec 



Teacher 

Administrator 

Stair 



Comments: 



Step m 



_ Mail 

Telephone 

In Class 

Visit 



Corajuents: 



Jan 

Feb 

Mar 

Apr 

May 

Jun 



_ Jul 
_ Aug 

Sep 

_ Oct 

Nov 

Dec 



Teacher 

Administrator 

StafT 



CAKOC/P R«seareh Stuily 



Page S 
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ReviMd January 20, 1989 



CAROC/P Research Study 
ROC/P Total Finances 



G<*neral Insi met ions 

Thi^ part of the survcj' ravers res'unue imd 
expenditure data for your entire ROC/P. PUm^^c 
report the ♦otal for all income and cxpundituri's 
conducted daring 1987-1988. Complete ail iU-mm on 
the attached form, writing "V/AT or "Unknown" on 
any lines for which daU is not available. Rc^rt 
unly thMe numbcra that apply to the 1987-1988 
reiM}xting yrar. 

ADA Instntctions 

Report the head-count enrollment , the Full Time 
E<iuivalcnt {PTE) Bverage daily attendant (ADA), 
and total number of student instructional hours for 
all courses. Entw concurrent and non-concurrent 
students 5q)aratc!y» the total the two g^ups on the 
line provided. 

Revcmieig Instructions 

We are interested in all revenues that were 
generated by this ROC/P. The categories to be 
reported follow the J-SOO-ROP form, except in acting 
for more detailed information In some areas. 

Please be careful to break out the "Sales From 
the Operation of a Business" and Toes & Cortracts* 
from the category "Other Local Revenues", Rq5f)rt 
only those other revenues that btc not sal .^ and/or 
fees on the Ifaie "All Other Local Revenue::"'. 

If there are iHher sources of revenue not 
accounted for in the regular catc^rics please include 
them on an '"Other" line. Detail the source trf each 
revenue amount, its EDP line number <if known), 
amd the amount eamoA It would be appreciated if 
you could also include a short explanation of what 
the revenue source is* how it was obtained, whether 
it is one time only or continuing, nd if there were 
restrictions on how it could be used. 



bcXiiil the total expenditures for this ROC/P. 
Indutle ail t^xpcndituri's made by this ROC/P, 
whether in dircvt suppi>rt of a particular a)urse or 
for iti'ms supporting multiple ctmrsejs Report all 
expenditures. 

Report teachers' salaries on a separate line firom 
all other ccrtiflcatcHi ^laries^ and separate 
Instructional aides* salaries from all other classified 
salaries^ 

Separate Total Services and Other Operating 
Expenses' by Tacilities\ ''Equipment", and "AU 
Other Services/Operating Expenses'. For both 
facilities and equipment check the appropriate Une 
and write in an amount deluding on whether it is: 

Free: available without any cost Enter tcro for cosL 
District: prwided for a fee by a Im»l school. The 

amount is the ann«irJ charge for this course* 
Rent: available thwjugh a I^se or rental agreement. 

Enter the annual amount paid for the rent. 
Own: owned by the ROC/P. The amount is the 

annual mortgage or bond paid If ftiUy paid for, 

enter r.era 

Detail other items of expenditure on an "Other" 
line. Total Expenditures* should be the sum of aU 
the lines, and nhould equal the total amount of 
expcn'lUurcs incurred by the ROC/P. Tlicre should 
be no excluded expenditures. 



Total Revenues" should be the sum of all the 
lines, and should equal the total amount of revenue 
generated by this ROC/P. There should bo no 
exduded incoma 



CAROC/P Re«earch Study 
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Revised January 20^ 1989 



ROC/P Total Finances 



During 19S7-198S, 



Concurrent: 

Non-Concurrent: 

Total: 



sections of all courses were taught at 

Enrollment ADA Stud Inst Hrs 



different locations. 



Revenues 



Actual Dollars (EDP) 

Total Revenue Limit Sources (200) 

Total Federal Revenues (280) 

Total Other State Revenues , . . (450) 

Total Other Local Revenues 

Sales From Operation of a Business (483) 

Fees & Contracts (508) 

All Other Locp.' Revenues 

Total Prior Year . venue Adjustment (001) 

Other: ( ) 

Othen ( ) 

Total Revenues , , (599) 



Expenditures 

Total Certificated Salaries 

Teachers' Salaries: . FTE 

All Other CertificateTSalaries: . FTE . . , 

Total Classified Salaries 

Instructional Aides* Salaries: . FTE 

All Other Classified Salaries: _ FTE 

Total Employee Benefits 

Total Books & Supplies 

Total Services and Other Operating ExpenEes 

Facilities: Free District Rent Owii 

Facilities Cost 

Equipment: Free District Rent Own 

Equipment Cost 

All Other Services/Operating Expenses 

Total Capital Outlay 

Total Other Outgo 

Total Direct Support/Indirect Costs 

Total Prior Year Expenditure & Other Adjust . . . . 

Other: .... 

Other. 

Total Expenditures 
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(600) 



(630) 



(729) 



(780) 
(846) 
(856) 
(002) 
( ) 
( ) 
(599) 



Rev!s»d Jamiaiy 198^ 
Unimndty of CaUfomi* mt Bhmfafa 



CAROC/P Research Study 
ROC/P Total Finances 

(1) Were any of the above income funds restricted in their use? YES NO 

If any were, please describe the source of the income, the amount restricted and 
what it was restricted to. 



(2) Were any of the above income funds from a single-use, temporary, or non-periodic 
source? YES NO 

If any were, please describe the source of the income, the amount uivolved, how 
many years (beyond 1987-1988) the monies would be available to your ROC/P, 
if the funds are renewable after that period, and if the use of the fund was 
restricted in any way. 



1 



(3) Did your ROC/P receive lottery funds during 1987-1988? YES NO 

If you did, describe how these funds were used (and if their use was restricted 
in any way). 



f 



CAROC/P Research Study 

^ ERfc 



167 



Page 8 



R«viae(l January 20, 1989 



CAROC/P Research Study 
ROC/P Total Finances 

(4) Were any of the expenditures given mandatory for your ROC/P? YES NO 

If any were, please describe tho object of the expenditure, the mandatory 
amount, who mandated the expenditure, and why it was mandatory. 



(5) Were any of the expenditures for objects of a single-use, temporary, or non-periodic 
source? YES NO 

If any were, please describe the object of the expenditure, the amount involved, 
how many years (beyond 1987-1988) the object would reoccur for your ROC/P, 
if the object is expected to continue after that period. 



(6) Were RO C/P employee's salaries negotiated by the ROC/P directly? YES NO 

If not, please describe how employee salaries are determined, and whether the 
ROC/P was required to hire under those salary limitations ^nly or if other 
structures could be used. 



168 



CABOC/P BMeareh Study 
^ . id imifa Ednntfamal BMnirch CeopmUve 



Reirised January 20» 19^ 



CAROC/P Research Study 
ROC/P Total Finances 



(7) Does your ROC/P have flexibility in choosing instructional sites ? YES NO 

If not, please describe the reasons that you use the sites that are used. Also, 
indicate if an equally suitable and less expensive site would be available 
otherwise, what it might cost, and how it would be used as an alternative. 



(8) Are the instructional equipment maintenance and replacement costs shared by the 
ROC/P with other agencies? YES NO 

If the costs are shared, please describe what pieces of equipment are shared, who 
they are shared with, and how much each entity pays for the upkeep of the 
equipment. 



(9) Do you make long-term provisions for the upgrading and/or replacement of ROC/P 
instructional equipment ? YES NO 

If you do, please describe the planning method used as well as how much money 
is periodically "saved" for this purpose. If no long-term planning is done, what 
provisions (if any) exist for the replacement of old and out-of-date equipment? 



(10) What was the ROC/P's Reserve Balance at the start of 1987-198S? t 
What was the ROC/P's Reserve Balance at the end of 1987-1988? $ 
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jRevised Jmnumry 20* 1989 



CAROC/P Research Study 
ROC/P Total Survey 



We are interested in any other pieces of information and comments that you would 
like to include as part of your responses. Please use the front and back sides of this page 
for any such comments, attaching addilional sheets of paper as desired. 



9 



f 
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Revised January 30, 1999 
Unlwnllr of Cftlifomfai si Riv^rsld* 



Appendix F: Cover Letter for the Survey Packets 



January 31, 1989 

{Director Name & Title] ^^^^ ^ ^ 

[ROC/P Name] 

[ROC/P Street Address] 

[ROC/P aty], [ROC/P State] [ROC/P Zip] 

Dear [Director Name], 

The California Educational Research Coopc ;itive (CERC), a research unit at 
the School of Education of the University of California at Riverside, has been 
rommissioned by the CaHfomia Association of Regional Occupational Centers and 
Rrc^rams (CAROC/P) to perform a research study. This study, underway since 
November of 1988, is looking at three specific areas concerning all ROC/Ps in the 
State of California. These areas are; 

(1) the legal, regulative, and fiscal history and development of ROC/Ps in 
California from their inception in 1963 through the present. 

(2) the relationship between inputs (costs and other program-related 
internal and external variables) and outputs (numbers of students 
enrolled, completed, and follow-up status) of selected ROC/P courses 

(3) the collection and use of information by the ROC/P for management 
decision making purposes (a prelude to the development of a 
management information system). 

We would appreciate your participation in the second phase of this study 
Enclosed wjt^ this letter you will find a site-level survey (requesting information 
on your ROC/P as a whole), from two to eight specific course-level surveys 
(requesting mformation on specific courses taught at your ROC/P), and a postage- 
^d return envelope. The specific courses selected have been chosen at random 
from each of eig^t different subject area domains (auto repair, off :e occupations, 
date processing, retailing, health occupations, quantity food, cosmetology^arberinff 
and CDnstniction), weighted by the size of your ROC/P, your specific courS 
offermgs, and the number of students In the different courses. 
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Appendix F: Cover Letter for the Survey Packets 



January 31, 1989 
Page 2 of 2 



Directions for completing the different surveys are included in with the 
actual survey sheets. We would appreciate if you could complete the surveys and 
return them all together in the postage-paid envelope provided. CERC Research 
Fellow Jeffrey Hecht is available at C714) 787-3026 if you or your staff should have 
any questions when completing the surveys. 

Every ROC/P that was operating in California during 1987-1988, including 
[ROC/P Name], is being sent a packet similar to the one you are now holding. To 
gain the most complete and accurate understanding of the costs of operating 
ROC/P we want to obtain feedback from as many sites as possible. Your 
cooperation is essential to the success of this effort . 

The anonymity of your responses will be upheld throughout all phases of 
this research. While each of the surveys is coded to your site such coding will be 
used only internally for data verification and cross-reference, not to be published 
as part of any interim or final reports. It is not the intent of this part of the 
stu4y to either commend top performer? or spotlight poor ones. Rather, we seek 
an understanding of what factors are both significant and under the control of the 
ROC/P in deciding course inputs and how those factors can be reliably related to 
specific course outcomes. Such information could prove to be an extremely useful 
tool for all ROC/Ps for future planning. 

Please return all of the surveys to us in the postage-paid envelope 
by the last day of February. This tight deadline is necessary if the CERC staff 
is to process and analyze all of the data in time for the Spring CAROC/P 
conference. Again, if there are any questions please do not hesitate to call. 

Thank you for your assistance. 



California Educational Research Cooperative 

School of Education 

University of California at Riverside 
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